Nonconvex Sets Research Articles

Conditional Gradient Method for Optimization Problems with a Constraint in the Form of the Intersection of a Convex Smooth Surface and a Convex Compact Set

The conditional gradient method is generalized to nonconvex sets of constraints representing the set-theoretic intersection of a convex smooth surface and a convex compact set. Necessary optimality conditions are studied, and the convergence of the method is analyzed.

Decentralized Riemannian Algorithm for Nonconvex Minimax Problems

The minimax optimization over Riemannian manifolds (possibly nonconvex constraints) has been actively applied to solve many problems, such as robust dimensionality reduction and deep neural networks with orthogonal weights (Stiefel manifold). Although many optimization algorithms for minimax problems have been developed in the Euclidean setting, it is difficult to convert them into Riemannian cases, and algorithms for nonconvex minimax problems with nonconvex constraints are even rare. On the other hand, to address the big data challenges, decentralized (serverless) training techniques have recently been emerging since they can reduce communications overhead and avoid the bottleneck problem on the server node. Nonetheless, the algorithm for decentralized Riemannian minimax problems has not been studied. In this paper, we study the distributed nonconvex-strongly-concave minimax optimization problem over the Stiefel manifold and propose both deterministic and stochastic minimax methods. The Steifel manifold is a non-convex set. The global function is represented as the finite sum of local functions. For the deterministic setting, we propose DRGDA and prove that our deterministic method achieves a gradient complexity of O( epsilon(-2)) under mild conditions. For the stochastic setting, we propose DRSGDA and prove that our stochastic method achieves a gradient complexity of O( epsilon(-4)). The DRGDA and DRSGDA are the first algorithms for distributed minimax optimization with nonconvex constraints with exact convergence. Extensive experimental results on the Deep Neural Networks (DNNs) training over the Stiefel manifold demonstrate the efficiency of our algorithms.

Hybrid zonotopes: A new set representation for reachability analysis of mixed logical dynamical systems

This article presents a new set representation named the hybrid zonotope that is equivalent to the union of 2N constrained zonotopes – convex polytopes – through the addition of N binary zonotope factors. The major contribution of this manuscript is a closed-form solution for exact forward reachable sets of discrete-time, linear hybrid systems modeled as mixed logical dynamical systems. The proposed approach captures the worst-case exponential growth in the number of convex sets required to represent the nonconvex reachable set while exhibiting only linear growth in the complexity of the hybrid zonotope set representation. Redundancy removal techniques are provided that leverage binary trees to store the combinations of binary factors of the hybrid zonotope that map to nonempty convex subsets. Numerical examples show the hybrid zonotope’s ability to compactly represent nonconvex reachable sets with an exponential number of features. Furthermore, the hybrid zonotope is shown to be closed under linear mappings, Minkowski sums, generalized intersections, and halfspace intersections.

Knowledge-Aided Federated Learning for Energy-Limited Wireless Networks

The conventional model aggregation-based federated learning (FL) approach requires all local models to have the same architecture, which fails to support practical scenarios with heterogeneous local models. Moreover, the frequent model exchange is costly for resource-limited wireless networks since modern deep neural networks usually have over a million parameters. To tackle these challenges, we first propose a novel knowledge-aided FL (KFL) framework, which aggregates light high-level data features, namely knowledge, in the per-round learning process. This framework allows devices to design their machine-learning models independently and reduces the communication overhead in the training process. We then theoretically analyze the convergence bound of the proposed framework under a non-convex loss function setting, revealing that scheduling more data volume in each round helps to improve the learning performance. In addition, large data volume should be scheduled in early rounds if the total scheduled data volume during the entire learning course is fixed. Inspired by this, we define a new objective function, i.e., the weighted scheduled data sample volume, to transform the inexplicit global loss minimization problem into a tractable one for device scheduling, bandwidth allocation, and power control. To deal with unknown time-varying wireless channels, we transform the considered problem into a deterministic problem for each round with the assistance of the Lyapunov optimization framework. Then, we derive the optimal bandwidth allocation and power control solution by convex optimization techniques. We also develop an efficient online device scheduling algorithm to achieve an energy-learning trade-off in the learning process. Experimental results on two typical datasets (i.e., MNIST and CIFAR-10) under highly heterogeneous local data distributions show that the proposed KFL is capable of reducing over 99% communication overhead while achieving better learning performance than the conventional model aggregation-based algorithms. In addition, the proposed device scheduling algorithm converges faster than the benchmark scheduling schemes.

On a numerical construction of doubly stochastic matrices with prescribed eigenvalues

We study the inverse eigenvalue problem for finding doubly stochastic matrices with specified eigenvalues. By making use of a combination of Dykstra’s algorithm and an alternating projection process onto a non-convex set, we derive hybrid algorithms for finding doubly stochastic matrices and symmetric doubly stochastic matrices with prescribed eigenvalues. Furthermore, we prove that the proposed algorithms converge, and linear convergence is also proved. Numerical examples are presented to demonstrate the efficiency of our method.

Constrained polynomial zonotopes

We introduce constrained polynomial zonotopes, a novel non-convex set representation that is closed under linear map, Minkowski sum, Cartesian product, convex hull, intersection, union, and quadratic as well as higher-order maps. We show that the computational complexity of the above-mentioned set operations for constrained polynomial zonotopes is at most polynomial in the representation size. The fact that constrained polynomial zonotopes are generalizations of zonotopes, polytopes, polynomial zonotopes, Taylor models, and ellipsoids further substantiates the relevance of this new set representation. In addition, the conversion from other set representations to constrained polynomial zonotopes is at most polynomial with respect to the dimension, and we present efficient methods for representation size reduction and for enclosing constrained polynomial zonotopes by simpler set representations.

Efficient Learning of a Linear Dynamical System With Stability Guarantees

We propose a principled method for projecting an arbitrary square matrix to the non-convex set of asymptotically stable matrices. Leveraging ideas from large deviations theory, we show that this projection is optimal in an information-theoretic sense and that it simply amounts to shifting the initial matrix by an optimal linear quadratic feedback gain, which can be computed exactly and highly efficiently by solving a standard linear quadratic regulator problem. The proposed approach allows us to learn the system matrix of a stable linear dynamical system from a single trajectory of correlated state observations. The resulting estimator is guaranteed to be stable and offers statistical bounds on the estimation error.

Stochastic variable metric proximal gradient with variance reduction for non-convex composite optimization

This paper introduces a novel algorithm, the Perturbed Proximal Preconditioned SPIDER algorithm (3P-SPIDER), designed to solve finite sum non-convex composite optimization. It is a stochastic Variable Metric Forward–Backward algorithm, which allows approximate preconditioned forward operator and uses a variable metric proximity operator as the backward operator; it also proposes a mini-batch strategy with variance reduction to address the finite sum setting. We show that 3P-SPIDER extends some Stochastic preconditioned Gradient Descent-based algorithms and some Incremental Expectation Maximization algorithms to composite optimization and to the case the forward operator can not be computed in closed form. We also provide an explicit control of convergence in expectation of 3P-SPIDER, and study its complexity in order to satisfy the approximate epsilon-stationary condition. Our results are the first to combine the non-convex composite optimization setting, a variance reduction technique to tackle the finite sum setting by using a minibatch strategy and, to allow deterministic or random approximations of the preconditioned forward operator. Finally, through an application to inference in a logistic regression model with random effects, we numerically compare 3P-SPIDER to other stochastic forward–backward algorithms and discuss the role of some design parameters of 3P-SPIDER.

有制約最適化のための制約条件の目的関数化と適応的重み調整を用いたMOEA/D

In this paper, MOEA/D is extended to constrained optimization by making the constraints an objective function. An adaptive adjustment method is proposed to introduce a parameter for varying weights. The parameter for varying the weights is given in such a way that the bias of the search towards the feasible and infeasible regions can be adjusted. The parameter is tuned based on two guidelines to properly utilize infeasible solutions. The first is actively utilizing infeasible solutions with large constraint violations and encouraging global search including the infeasible regions. The second is actively utilizing infeasible solutions with small constraint violations and encouraging a search on the boundary of the feasible regions. This is expected to improve the global optimization performance to the feasible regions, which is a non-convex set, and the convergence performance to feasible solutions. We verify the usefulness of the proposed method for problems where the feasible regions are a convex set and a nonconvex set.

Chronos: Accelerating Federated Learning With Resource Aware Training Volume Tuning at Network Edges

Due to the limited resources and data privacy issue, last decade witnesses the fast development of Distributed Machine Learning (DML) at network edges. Among all the existing DML paradigms, Federated Learning (FL) would be a promising one, since in FL, each client trains its local model without sharing the raw data with others. A community of clients with the same interest can join together to derive a high-performance model by periodically synchronizing the parameters of their local models under the help of a coordination server. However, FL will encounter the straggler problem at network edges, and hence the synchronization among clients becomes inefficient. It slows down the convergence speed of learning process. To alleviate the straggler problem, we propose a method named Chronos that accelerates FL with training volume tuning in this paper. More specifically, Chronos is a resource aware method that adaptively adjusts the amount of data used by each client for training (i.e., training volume) in each iteration in order to eliminate the synchronization waiting time caused by the heterogeneous and dynamical computing and communication resources. In addition, we theoretically analyze the convergence of Chronos in a non-convex setting and utilize the results for the algorithm design of Chronos in return to guarantee the convergence. Extensive experiments show that compared with the benchmark algorithms (i.e., BSP and SSP), Chronos significantly improves convergence speed by up to 6.4×.

Low-Latency Federated Learning With DNN Partition in Distributed Industrial IoT Networks

Federated Learning (FL) empowers Industrial Internet of Things (IIoT) with distributed intelligence of industrial automation thanks to its capability of distributed machine learning without any raw data exchange. However, it is rather challenging for lightweight IIoT devices to perform computation-intensive local model training over large-scale deep neural networks (DNNs). Driven by this issue, we develop a communication-computation efficient FL framework for resource-limited IIoT networks that integrates DNN partition technique into the standard FL mechanism, wherein IIoT devices perform local model training over the bottom layers of the objective DNN, and offload the top layers to the edge gateway side. Considering imbalanced data distribution, we derive the device-specific participation rate to involve the devices with better data distribution in more communication rounds. Upon deriving the device-specific participation rate, we propose to minimize the training delay under the constraints of device-specific participation rate, energy consumption and memory usage. To this end, we formulate a joint optimization problem of device scheduling and resource allocation (i.e. DNN partition point, channel assignment, transmit power, and computation frequency), and solve the long-term min-max mixed integer non-linear programming based on the Lyapunov technique. In particular, the proposed dynamic device scheduling and resource allocation (DDSRA) algorithm can achieve a trade-off to balance the training delay minimization and FL performance. We also provide the FL convergence bound for the DDSRA algorithm with both convex and non-convex settings. Experimental results demonstrate the derived device-specific participation rate in terms of feasibility, and show that the DDSRA algorithm outperforms baselines in terms of test accuracy and convergence time.

Status quo-dependent user equilibrium model with adaptive value of time

Based on the status quo-dependent route choice model in Xu et al. (2017), this study encapsulates the route choice model into traffic assignment modeling and establishes a Status quo-dependent User Equilibrium (SDUE) model. The proposed SDUE model adopts the adaptive value of time (VOT) to incorporate three kinds of route choice behavior: cognitive limitations and capability constraints, satisficing behavior and inertial behavior, and variation in VOTs, which is further compared with existing UE models. The connection between the SDUE model and existing UE models is analyzed. It is theoretically proved that equilibrium solutions from previous UE models can all be included in the SDUE solution set by letting VOTs vary among users and across scenarios. Moreover, a reduced SDUE model is formulated to help calculate possible SDUE flow patterns mathematically. Numerical examples are given to illustrate that the proposed SDUE model could include the non-convex solution set and various route choice behavior by adjusting the adaptive VOT-related parameters, which is beyond the existing UE models. The non-uniqueness of the SDUE solutions and model applications are discussed.

A class of differential hemivariational inequalities constrained on nonconvex star-shaped sets

The purpose of this paper is to investigate a class of nonconvex-constrained differential hemivariational inequalities consisting of nonlinear evolution equations and evolutionary hemivariational inequalities. The admissible set of constraints is closed and star-shaped with respect to a certain ball in a reflexive Banach space. We construct an auxiliary inclusion problem and obtain the existence results by applying a surjectivity theorem for multivalued pseudomonotone operators and the properties of Clarke subgradient operator. Moreover, the existence of a solution to the original problem is established by hemivariational inequality approach and a penalization method in which a small parameter does not have to tend to zero. Finally, an application of the main results is provided.

Stochastically Controlled Compositional Gradient for Composition Problems.

We consider composition problems of the form (1/n)∑i = 1n Fi (1/m)∑j = 1m Gj(x) , which are important for machine learning. Although gradient descent and stochastic gradient descent are straightforward solutions, the essential computation of G (x) = (1/m)∑j = 1mGj(x) in each single iteration is expensive, let alone for large m . In this article, we devise a stochastically controlled compositional gradient algorithm. Specifically, we introduce two variants of stochastically controlled technique to estimate the inner function G(x) and the gradient of the objective function, respectively. The computational cost is largely reduced. However, the natural needs of two stochastic subsets D1 and D2 form direct barriers to guarantee the convergence of the algorithm, especially the theoretical proof of the convergence. To this end, we present a general convergence analysis by proving | D1|=min{1/ϵ,m} and | D2|=min{1/ϵ,n } , through which the proposed method significantly improve composition algorithms under low target accuracy (i.e., 1/ϵ << m or n ) in both strongly convex and nonconvex settings. Comprehensive experiments demonstrate the superiority of the proposed method over existing methods.

Strict weak l-efficient solutions for nonconvex set optimization problems

In this article, we introduce the notions of l-transfer lower continuous and q-level intersectionally closed for set-valued mappings with respect to the lower set less relation. Then, we obtain some existence results for strict weak l-efficient solutions of such set-valued mappings. Moreover, we prove some existence results for nonconvex set optimization problems via asymptotic analysis tools, in the setting of the Banach spaces equipped with a Hausdorff topology σ coarser than the norm topology.

Last-iterate convergence analysis of stochastic momentum methods for neural networks

The stochastic momentum method is a commonly used acceleration technique for solving large-scale stochastic optimization problems. Current convergence results of stochastic momentum methods under non-convex stochastic settings mostly discuss convergence in terms of the random output and minimum output, which requires temporal and spatial statistics of historical data. On the other hand, the last-iterate convergence allows us to avoid storing or selecting past output iterates after each iteration, while maintaining rigour in convergence analysis. To this end, we address the convergence of the last iterate output (called last-iterate convergence) of the stochastic momentum methods for non-convex stochastic optimization problems, in a way which is conformal with traditional optimization theory. For generality, we prove the last-iterate convergence of the stochastic momentum methods under a unified framework, covering both stochastic heavy ball momentum and stochastic Nesterov accelerated gradient momentum, whose momentum factors can be either constant or time-varying coefficients. Finally, the last-iterate convergence of the stochastic momentum methods is verified on the benchmark MNIST and CIFAR-10 datasets. The implementation of SUM is available at: https://github.com/xudp100/SUMhttps://github.com/xudp100/SUM.

Formation Coordination Control of Leaderless Multi-AUV System with Double Independent Communication Topology and Nonconvex Control Input Constraints

In this paper, the formation coordination control of discrete-time distributed leaderless multiple autonomous underwater vehicle (AUV) system with double independent position–velocity communication topology and control inputs on a nonconvex set is studied. Firstly, the problem of formation coordination control of multi-AUV system is transformed into the problem of formation consensus of multi-AUV system, and the consistent state of leaderless multi-AUV system formation was defined. Secondly, considering the existence of bounded communication delay and nonconvex control input constraints for multi-AUV system formation under weak communication conditions, a formation consistent constraint controller algorithm for discrete-time leaderless multi-AUV system with double independent communication topology is proposed by introducing constraint operators. By using the properties of graph theory, random matrix and SIA matrix, and selecting appropriate controller parameters, the multi-AUV system formation can reach the defined consensus state. Furthermore, the unbounded communication delay of multi-AUV system formation is studied. Finally, the simulation results show that the proposed controller constraint algorithm is effective.

Industrial Edge Intelligence: Federated-Meta Learning Framework for Few-Shot Fault Diagnosis

The scarcity of fault samples has been the bottleneck for the large-scale application of mechanical fault diagnosis (FD) methods in the industrial Internet of Things (IIoT). Traditional few-shot FD methods are fundamentally limited in that the models can only learn from the direct dataset, i.e., a limited number of local data samples. Federated learning (FL) has recently shown the capacity of collaborative artificial intelligence and privacy preservation. Based on these capabilities, we propose a novel approach to solve the few-shot FD problem, which includes a generic framework (i.e., FedMeta-FFD) and an easy-to-implement enhancement technique (i.e., AILR). The FedMeta-FFD framework allows clients to learn from indirect datasets owned by other collaborators while training a global meta-learner to solve the few-shot problem directly. More concretely, with only a few labeled examples and training iterations, the global meta-learner can quickly adapt to a new client (e.g., a machine under different operating conditions) or a newly encountered fault category. Adopting AILR can significantly improve the performance of the FedMeta-FFD framework while also increasing the stability of the learning process. Further, we conduct a theoretical analysis of the proposed framework's convergence in a non-convex setting. We thoroughly evaluate the proposed FedMeta-FFD on two fault diagnosis datasets and also perform the practical validation on real-world IIoT scenarios. They demonstrate that our proposed approach achieves significantly faster convergence and higher accuracy than the existing approaches.

On the structure of the singularity of the solution to the time-optimal control problem in the case of a discontinuity in the curvature of the boundary of the target set

A time-optimal control problem in two-dimensional space with a spherical velocity vectogram and a non-convex target set with a boundary having a finite number of points of discontinuity in the curvature is considered. Identification and construction of scattering curves that form a singular set of the optimal result function are studied for the case when the points of discontinuity in the curvature have one-sided curvatures of different sign. It is shown that these points belong to pseudo-vertices that are characteristic points of the target set boundary responsible for generation of singular set branches. The structure of scattering curves and optimal trajectories starting from them, which fall in the neighborhood of the pseudo-vertex, is investigated. A characteristic feature of the case under study is revealed. It is proved that one pseudo-vertex can generate two different singular set branches. The results obtained are illustrated by an example of solving the control problem for the target set as a one-dimensional manifold.

Integral Design Method for Control Barrier Functions that Considers the Shape of the Target System

In recent years, control barrier functions have attracted much attention for ensuring safety in collision avoidance and human assist control. However, control laws cannot guarantee the solution of differential equations without properly considering the shape of the target system. In this study, we designed a control law that considers the shape of the control target and designs a control barrier function for non-convex safe sets. Based on this, we proposed an integral control barrier function and confirmed its effectiveness by testing it for the collision avoidance between a two-links robot arm and an obstacle through simulations.