Frank-Wolfe Research Articles

Avoiding bad steps in Frank-Wolfe variants

The study of Frank-Wolfe (FW) variants is often complicated by the presence of different kinds of “good” and “bad” steps. In this article, we aim to simplify the convergence analysis of specific variants by getting rid of such a distinction between steps, and to improve existing rates by ensuring a non-trivial bound at each iteration. In order to do this, we define the Short Step Chain (SSC) procedure, which skips gradient computations in consecutive short steps until proper conditions are satisfied. This algorithmic tool allows us to give a unified analysis and converge rates in the general smooth non convex setting, as well as a linear convergence rate under a Kurdyka-Łojasiewicz (KL) property. While the KL setting has been widely studied for proximal gradient type methods, to our knowledge, it has never been analyzed before for the Frank-Wolfe variants considered in the paper. An angle condition, ensuring that the directions selected by the methods have the steepest slope possible up to a constant, is used to carry out our analysis. We prove that such a condition is satisfied, when considering minimization problems over a polytope, by the away step Frank-Wolfe (AFW), the pairwise Frank-Wolfe (PFW), and the Frank-Wolfe method with in face directions (FDFW).

Political and Institutional Approaches to Advancing International Religious Freedom in the U.S. Foreign Policy in 1998–2020

The article traces the institutional and political evolution of the US implementing its concept of international religious freedom from 1998 to 2020. The concept became popular after the end of the Cold War when the defense of religion against socialistatheist ideas had lost its relevance due to political (the collapse of the Eastern bloc) and cultural (the growing importance of religion in many non-Western countries of the world) reasons. The article starts by reviewing the history of the emergence and development of the institutional infrastructure (legal acts, administrative entities, public and political organizations, think tanks, expert platforms, interdepartmental coordination formats, human rights projects, funding and grant programs, and new international organizations) that supported the US foreign policy advancing international religious freedom. The article divides the reconstruction of this process into two periods formed by adopting two acts: in 1998 — the International Religious Freedom Act, and in 2016 — the amendment to this Act named after congressman Frank Wolf. The study concludes that over the past twenty years, religion has finally developed as an independent track of the US foreign policy, although it has been developing inconsistently and has undergone repeated corrections. The concept of international religious freedom has become the ideational basis of this track, while the extensive institutional and political infrastructure serves as its apparatus. More than 80 percent of the world's population encounters religious persecution. It legitimizes the concept, which encourages widespread support for religious minorities as opposed to the religious majority, and opens the way for the US to build sustainable ties with religious communities and their leaders worldwide, allowing them to influence political processes in different regions. The US has recently promoted the integration of this policy track into international institutions, establishing the International Religious Freedom or Belief Alliance.

A bi-level model and hybrid heuristic algorithm for the optimal location of prefabricated building industrial park

Optimal location for prefabricated building industrial park (PBIP) can significantly reduce the logistic cost, delivery time and environmental pollution of prefabricated and modular construction. PBIP site selection problems can be more complex than other location selection problems when considering the goals of the PBIP at a manager level and a costumer level. There is little to no research that has examined the bi-level model for PBIP site selection. Thus, this paper addresses this issue by constructing a quantitative bi-level programming model which considers the economic, traffic, and environmental constrains. Subsequently, combining the genetic algorithm and Partan Frank–Wolfe​ algorithm, a hybrid heuristic algorithm is constructed to calculate the model. An empirical analysis is then carried out in the case of Chongqing, China. The results clearly demonstrate the applicability of the model. Furthermore, the paper discusses the trade-off between transport costs and environmental benefits, and the impact of transport radius on transport schemes. The proposed model contributes to solving location–allocation problems. It considers road congestion and traffic distribution by combining the location–allocation model with the user equilibrium model. The model can aid planners in making more robust decisions for setting up new PBIPs, thus contributing to the sustainable development of prefabricated and modular construction.

A stochastic conditional gradient algorithm for decentralized online convex optimization

We study in this paper several problems at the intersection of decentralized optimization and online learning. Decentralized optimization plays a vital role in machine learning and has recently garnered much attention due to its inherent advantage in handling edge computations. Many decentralized optimization algorithms, both projection and projection-free algorithms with theoretical guarantees, have been proposed in the literature, focusing mainly on offline settings. However, for most real-world machine learning problems, the data is often revealed online, for example, in the case of recommender systems, image/video processing, and stock portfolio management. Therefore, in this work, we study decentralized optimization within the framework of online settings with constraints imposed on the optimization solutions (e.g., sparsity or low rank of matrices). More specifically, we consider the problem of optimizing an aggregate of convex loss functions that arrive over time such that their components are distributed over a connected network. We present a consensus-based online decentralized Frank-Wolfe algorithm that uses stochastic gradient estimates, which achieves an asymptotically tight regret guarantee of O(T) where T is a given time horizon. Furthermore, we demonstrate the performance of this algorithm for optimizing the online multiclass logistic regression model on real-world standard image datasets (MNIST, CIFAR10) by comparing with centralized online algorithms. We achieve better regret bounds than the previously best-known decentralized constrained online algorithms.

Distributed hierarchical control for multiple refrigeration units

This paper presents a hierarchical distributed control strategy for multiple refrigeration units (RUs). The main target is to create a modular architecture that provides the thermal demand of the central plant in a multi-agent framework. Combined with the model predictive control method, a robust and effective framework is developed that aims to minimize the energy consumption while reducing latency times and switching of the RUs. The upper layer model predictive control is identical for all RUs, while in a lower layer a mixed-integer optimization using the decentralized Frank–Wolfe algorithm secures optimal switching of the RUs. The operational behavior can be easily and transparently adjusted in a wide range. Furthermore, availability and quick replacement of faulty units is guaranteed by the concept. A simulation example consisting of an insulated cool box on a truck demonstrates the versatility and performance of the proposed approach. • Distributed hierarchical optimization for multiple refrigeration units is presented. • Model predictive control with mixed-integer optimization is employed. • Multiple refrigeration units maximize availability for the central plant. • Transparent weighting factors enable wide variety of operating patterns.

Threshold-free multi-attributes physical layer authentication based on expectation–conditional maximization channel estimation in Internet of Things

With the number of Internet of Things devices continually increasing, the endogenous security of Internet of Things communication systems is growingly critical. Physical layer authentication is a powerful means of resisting active attacks by exploiting the unique characteristics inherent in wireless signals and physical devices. Many existing physical layer authentication schemes usually assume physical layer attributes obey certain statistical distributions that are unknown to receivers. To overcome the uncertainty, machine learning–based authentication approaches have been employed to implement threshold-free authentication. In this article, we utilize an expectation–conditional maximization algorithm to provide the physical layer attribute estimates required for the authentication phase and a logistic regression model to achieve threshold-free physical layer authentication. Moreover, a Frank–Wolfe algorithm is considered to achieve fast convergence of the logistic regression parameters and multi-attributes are adopted to increase the differentiation of transmitters. Simulation results demonstrate that the obtained attribute estimates are sufficient to provide a reliable source of data for authentication and the proposed threshold-free multi-attributes physical layer authentication scheme can effectively improve authentication accuracy, with the false alarm rate P f reduced to 0.0263% and the miss detection rate P m reduced to 0.3466%.

Flow-Capture Location Model with Link Capacity Constraint Over a Mixed Traffic Network

Abstract This paper constructs and settles a charging facility location problem with the link capacity constraint over a mixed traffic network. The reason for studying this problem is that link capacity constraint is mostly insufficient or missing in the studies of traditional user equilibrium models, thereby resulting in the ambiguous of the definition of road traffic network status. Adding capacity constraints to the road network is a compromise to enhance the reality of the traditional equilibrium model. In this paper, we provide a two-layer model for evaluating the efficiency of the charging facilities under the condition of considering the link capacity constraint. The upper level model in the proposed bi-level model is a nonlinear integer programming formulation, which aims to maximize the captured link flows of the battery electric vehicles. Moreover, the lower level model is a typical traffic equilibrium assignment model except that it contains the link capacity constraint and driving distance constraint of the electric vehicles over the mixed road network. Based on the Frank-Wolfe algorithm, a modified algorithm framework is adopted for solving the constructed problem, and finally, a numerical example is presented to verify the proposed model and solution algorithm.

Screening for a Reweighted Penalized Conditional Gradient Method

The conditional gradient method (CGM) is widely used in large-scale sparse convex optimization, having a low per iteration computational cost for structured sparse regularizers and a greedy approach for collecting nonzeros. We explore the sparsity acquiring properties of a general penalized CGM (P-CGM) for convex regularizers and a reweighted penalized CGM (RP-CGM) for nonconvex regularizers, replacing the usual convex constraints with gauge-inspired penalties. This generalization does not increase the per-iteration complexity noticeably. Without assuming bounded iterates or using line search, we show O(1/t) convergence of the gap of each subproblem, which measures distance to a stationary point. We couple this with a screening rule which is safe in the convex case, converging to the true support at a rate O(1/(δ 2 )) where δ≥0 measures how close the problem is to degeneracy. In the nonconvex case the screening rule converges to the true support in a finite number of iterations, but is not necessarily safe in the intermediate iterates. In our experiments, we verify the consistency of the method and adjust the aggressiveness of the screening rule by tuning the concavity of the regularizer.

Target encirclement of moving ride-hailing vehicle under uncertain environment: A multi-vehicle mutual rescue model

To solve the emergency rescue of driver-hijacking-passengers event, this paper proposes a multi-vehicle mutual rescue (MMR) model for ride-hailing platform to solve the emergency rescue of driver-hijacking-passengers event. Firstly, the upper model is constructed minimize the movable link lengths of the target vehicle, the path risk of rescue vehicles according to robust optimization theory. Secondly, the lower model is formed by the user equilibrium (UE) model for traffic assignment problems by using mutual and positioning information. Thirdly, the strength Pareto evolutionary algorithm (SPEA Ⅱ), the Frank-Wolfe algorithm, and the method of two-stage coding are comprehensively applied in this paper for the NP-hard problem of the MMR model. Finally, the mutual information and O-D flows of links on the 4th Ring Road of Beijing are utilized to examine the MMR model, and analyzed the utility of the hybrid algorithm and the MMR model. The experimental result indicates that when the robust control parameters are set to 0, 30, and 60, then 3, 2, and 3 groups of the MMR schemes with different sensitivity to the uncertain environment can be obtained. The algorithm utility analysis results indicate that the hybrid algorithm with intimacy calculation can significantly enhance the convergence of the algorithm. Compared with the traditional convex hull algorithm, the model utility analysis results indicate that, in addition to effectively making up for the application defects of the traditional algorithm, the MMR model adopts the mutual rescue strategy that is superior to the traditional algorithm in the Target encirclement nodes selection of target vehicle, the paths planning of rescue vehicles, the collaborative scheduling of ride-hailing platform, and can well complete the mutual rescue task under an uncertain environment for driver-hijacking-passengers event.

Optimal probing with statistical guarantees for network monitoring at scale

Monitoring large-scale cloud networks is a complex task because their scale is prohibitively large, monitoring budgets are limited, network topologies are not regular and the estimates produced are a function of traffic patterns. In this work, we take a statistical approach to estimating a network metric, such as the latency of a set of paths, with guarantees on the estimation error. We aim to do so in an intelligent and scalable manner, without observing all existing traffic, and minimizing the estimation error at a fixed probing budget per unit of time. Our algorithms produce a distribution of probes/samples across network paths which can be used in conjunction with existing probers (or samplers). These algorithms are based on A- and E-optimal experimental designs in statistics, which guarantee a bounded estimation error for any monitoring budget. Unfortunately, these designs are too computationally intensive to be used in production at scale. We propose a scalable and near-optimal approximate implementations based on the Frank–Wolfe algorithm. We validate our approaches with two metrics (latency and loss) in simulations on real network topologies, and also using a production probing system in a real cloud network. We show major gains in reducing the probing budget compared to both production and academic baselines, while maintaining low errors in estimates, even with very low probing budgets.

On the primal and dual formulations of traffic assignment problems with perception stochasticity and demand elasticity

ABSTRACT This article reinvestigates the mathematical formulations of traffic assignment problems with perception stochasticity and demand elasticity in both the system optimum and user equilibrium principles. Our focus is given to a pair of new general formulations that pose a duality relationship to each other. In this primal-dual modeling framework, we found that the equilibrium or optimality conditions of a traffic assignment problem with perception stochasticity and demand elasticity can be redefined as a combination of three sets of equations and an arbitrary feasible solution of either the primal or dual formulation satisfies only two of them. We further rigorously proved the solution equivalency and uniqueness of both the primal and dual formulations, by using derivative-based techniques. While the two formulations pose their respective modeling advantages and drawbacks, our preliminary algorithmic analysis and numerical test results indicate that the dual formulation-based algorithm, i.e., the Cauchy algorithm, can be more readily implemented for large-scale problems and converge evidently faster than the primal formulation-based one, i.e. the Frank-Wolfe algorithm.

Code and Data Repository for FrankWolfe.jl: A High-performance and Flexible Toolbox for Frank-Wolfe Algorithms and Conditional Gradients

The software and data in this repository are a snapshot of the software and data that were used in the research reported on in the paper FrankWolfe.jl: a high-performance and flexible toolbox for Frank-Wolfe algorithms and Conditional Gradients by M. Besançon, A. Carderera, and S. Pokutta.

Нижние оценки для методов типа условного градиента для задач минимизации гладких сильно выпуклых функций

In this paper, we consider conditional gradient methods. These are methods that use a linear minimization oracle, which, for a given vector $p \in \mathbb{R}^n$, computes the solution of the subproblem $$\arg \min_{x\in X}{\langle p,x \rangle}.$$ There are a variety of conditional gradient methods that have a linear convergence rate in a strongly convex case. However, in all these methods, the dimension of the problem is included in the rate of convergence, which in modern applications can be very large. In this paper, we prove that in the strongly convex case, the convergence rate of the conditional gradient methods in the best case depends on the dimension of the problem $ n $ as $ \widetilde {\Omega} (\sqrt {n}) $. Thus, the conditional gradient methods may turn out to be ineffective for solving strongly convex optimization problems of large dimensions. Also, the application of conditional gradient methods to minimization problems of a quadratic form is considered. The effectiveness of the Frank-Wolfe method for solving the quadratic optimization problem in the convex case on a simplex (PageRank) has already been proved. This work shows that the use of conditional gradient methods to solve the minimization problem of a quadratic form in a strongly convex case is ineffective due to the presence of dimension in the convergence rate of these methods. Therefore, the Shrinking Conditional Gradient method is considered. Its difference from the conditional gradient methods is that it uses a modified linear minimization oracle. The convergence rate of such an algorithm does not depend on dimension. Using the Shrinking Conditional Gradient method the complexity of solving the minimization problem of quadratic form on a $ \infty $-ball is obtained. The resulting evaluation of the method is comparable to the complexity of the gradient method.

A Generalized Conditional Gradient Method for Dynamic Inverse Problems with Optimal Transport Regularization

We develop a dynamic generalized conditional gradient method (DGCG) for dynamic inverse problems with optimal transport regularization. We consider the framework introduced in Bredies and Fanzon (ESAIM: M2AN 54:2351–2382, 2020), where the objective functional is comprised of a fidelity term, penalizing the pointwise in time discrepancy between the observation and the unknown in time-varying Hilbert spaces, and a regularizer keeping track of the dynamics, given by the Benamou–Brenier energy constrained via the homogeneous continuity equation. Employing the characterization of the extremal points of the Benamou–Brenier energy (Bredies et al. in Bull Lond Math Soc 53(5):1436–1452, 2021), we define the atoms of the problem as measures concentrated on absolutely continuous curves in the domain. We propose a dynamic generalization of a conditional gradient method that consists of iteratively adding suitably chosen atoms to the current sparse iterate, and subsequently optimizing the coefficients in the resulting linear combination. We prove that the method converges with a sublinear rate to a minimizer of the objective functional. Additionally, we propose heuristic strategies and acceleration steps that allow to implement the algorithm efficiently. Finally, we provide numerical examples that demonstrate the effectiveness of our algorithm and model in reconstructing heavily undersampled dynamic data, together with the presence of noise.

Fast Cluster Detection in Networks by First Order Optimization

Cluster detection plays a fundamental role in the analysis of data. In this paper, we focus on the use of s-defective clique models for network-based cluster detection and propose a nonlinear optimization approach that efficiently handles those models in practice. In particular, we introduce an equivalent continuous formulation for the problem under analysis, and we analyze some tailored variants of the Frank-Wolfe algorithm that enable us to quickly find maximal s-defective cliques. The good practical behavior of those algorithmic tools, which is closely connected to their support identification properties, makes them very appealing in practical applications. The reported numerical results clearly show the effectiveness of the proposed approach.

Space planning considering congestion in container terminal yards

This paper proposes a nonlinear mathematical model for solving the problem of allocating storage space to future container arrivals in container terminal yards by considering the waiting times of trucks in the yard, called stochastic space allocation model (SSAM). Three heuristic procedures based on the Frank-Wolfe algorithm and the subgradient optimization method are proposed to efficiently solve the problem. From the numerical experiment, it is found that the proposed algorithms for SSAM provide solutions with the gaps between the objective values and the optimal values are no greater than 0.69%. It was also found that the best algorithm among three proposed algorithms (FWMP-S) obtained the average 7.89% improvement in the objective value compared with a previous model with a handling capacity constraint instead of considering waiting time (DSAM). It is found that the SSAM reduces the average system time of transporters for ship operation and that of road trucks by 17.39 and 7.14%, respectively, compared with the DSAM. This study proposes the most appropriate heuristic algorithm depending on the problem characteristics of the SSAM.

Perturbed Fenchel duality and first-order methods

We show that the iterates generated by a generic first-order meta-algorithm satisfy a canonical perturbed Fenchel duality inequality. The latter in turn readily yields a unified derivation of the best known convergence rates for various popular first-order algorithms including the conditional gradient method as well as the main kinds of Bregman proximal methods: subgradient, gradient, fast gradient, and universal gradient methods.

Restarting Frank–Wolfe: Faster Rates under Hölderian Error Bounds

Conditional gradient algorithms (aka Frank–Wolfe algorithms) form a classical set of methods for constrained smooth convex minimization due to their simplicity, the absence of projection steps, and competitive numerical performance. While the vanilla Frank–Wolfe algorithm only ensures a worst-case rate of $${\mathcal {O}}(1/\epsilon )$$ , various recent results have shown that for strongly convex functions on polytopes, the method can be slightly modified to achieve linear convergence. However, this still leaves a huge gap between sublinear $${\mathcal {O}}(1/\epsilon )$$ convergence and linear $${\mathcal {O}}(\log 1/\epsilon )$$ convergence to reach an $$\epsilon $$ -approximate solution. Here, we present a new variant of conditional gradient algorithms that can dynamically adapt to the function’s geometric properties using restarts and smoothly interpolates between the sublinear and linear regimes. These interpolated convergence rates are obtained when the optimization problem satisfies a new type of error bounds, which we call strong Wolfe primal bounds. They combine geometric information on the constraint set with Hölderian error bounds on the objective function.

Generalized self-concordant analysis of Frank–Wolfe algorithms

Projection-free optimization via different variants of the Frank–Wolfe method has become one of the cornerstones of large scale optimization for machine learning and computational statistics. Numerous applications within these fields involve the minimization of functions with self-concordance like properties. Such generalized self-concordant functions do not necessarily feature a Lipschitz continuous gradient, nor are they strongly convex, making them a challenging class of functions for first-order methods. Indeed, in a number of applications, such as inverse covariance estimation or distance-weighted discrimination problems in binary classification, the loss is given by a generalized self-concordant function having potentially unbounded curvature. For such problems projection-free minimization methods have no theoretical convergence guarantee. This paper closes this apparent gap in the literature by developing provably convergent Frank–Wolfe algorithms with standard mathcal {O}(1/k) convergence rate guarantees. Based on these new insights, we show how these sublinearly convergent methods can be accelerated to yield linearly convergent projection-free methods, by either relying on the availability of a local liner minimization oracle, or a suitable modification of the away-step Frank–Wolfe method.

Improved complexities for stochastic conditional gradient methods under interpolation-like conditions

We analyze stochastic conditional gradient methods for constrained optimization problems arising in over-parametrized machine learning. We show that one could leverage the interpolation-like conditions satisfied by such models to obtain improved oracle complexities. Specifically, when the objective function is convex, we show that the conditional gradient method requires O(ϵ−2) calls to the stochastic gradient oracle to find an ϵ-optimal solution. Furthermore, by including a gradient sliding step, we show that the number of calls reduces to O(ϵ−1.5).