Nonconvex Sets Research Articles

Industrial Edge Intelligence: Federated-Meta Learning Framework for Few-Shot Fault Diagnosis

The scarcity of fault samples has been the bottleneck for the large-scale application of mechanical fault diagnosis (FD) methods in the industrial Internet of Things (IIoT). Traditional few-shot FD methods are fundamentally limited in that the models can only learn from the direct dataset, i.e., a limited number of local data samples. Federated learning (FL) has recently shown the capacity of collaborative artificial intelligence and privacy preservation. Based on these capabilities, we propose a novel approach to solve the few-shot FD problem, which includes a generic framework (i.e., FedMeta-FFD) and an easy-to-implement enhancement technique (i.e., AILR). The FedMeta-FFD framework allows clients to learn from indirect datasets owned by other collaborators while training a global meta-learner to solve the few-shot problem directly. More concretely, with only a few labeled examples and training iterations, the global meta-learner can quickly adapt to a new client (e.g., a machine under different operating conditions) or a newly encountered fault category. Adopting AILR can significantly improve the performance of the FedMeta-FFD framework while also increasing the stability of the learning process. Further, we conduct a theoretical analysis of the proposed framework's convergence in a non-convex setting. We thoroughly evaluate the proposed FedMeta-FFD on two fault diagnosis datasets and also perform the practical validation on real-world IIoT scenarios. They demonstrate that our proposed approach achieves significantly faster convergence and higher accuracy than the existing approaches.

On the structure of the singularity of the solution to the time-optimal control problem in the case of a discontinuity in the curvature of the boundary of the target set

A time-optimal control problem in two-dimensional space with a spherical velocity vectogram and a non-convex target set with a boundary having a finite number of points of discontinuity in the curvature is considered. Identification and construction of scattering curves that form a singular set of the optimal result function are studied for the case when the points of discontinuity in the curvature have one-sided curvatures of different sign. It is shown that these points belong to pseudo-vertices that are characteristic points of the target set boundary responsible for generation of singular set branches. The structure of scattering curves and optimal trajectories starting from them, which fall in the neighborhood of the pseudo-vertex, is investigated. A characteristic feature of the case under study is revealed. It is proved that one pseudo-vertex can generate two different singular set branches. The results obtained are illustrated by an example of solving the control problem for the target set as a one-dimensional manifold.

Integral Design Method for Control Barrier Functions that Considers the Shape of the Target System

In recent years, control barrier functions have attracted much attention for ensuring safety in collision avoidance and human assist control. However, control laws cannot guarantee the solution of differential equations without properly considering the shape of the target system. In this study, we designed a control law that considers the shape of the control target and designs a control barrier function for non-convex safe sets. Based on this, we proposed an integral control barrier function and confirmed its effectiveness by testing it for the collision avoidance between a two-links robot arm and an obstacle through simulations.

Robust Adaptive Beamforming via Worst-Case SINR Maximization With Nonconvex Uncertainty Sets

This paper considers a formulation of the robust adaptive beamforming (RAB) problem based on worst-case signal-to-interference-plus-noise ratio (SINR) maximization with a nonconvex uncertainty set for the steering vectors. The uncertainty set consists of a similarity constraint and a (nonconvex) double-sided ball constraint. The worst-case SINR maximization problem is turned into a quadratic matrix inequality (QMI) problem using the strong duality of semidefinite programming. Then a linear matrix inequality (LMI) relaxation for the QMI problem is proposed, with an additional valid linear constraint. Necessary and sufficient conditions for the tightened LMI relaxation problem to have a rank-one solution are established. When the tightened LMI relaxation problem still has a high-rank solution, the LMI relaxation problem is further restricted to become a bilinear matrix inequality (BLMI) problem. We then propose an iterative algorithm to solve the BLMI problem that finds an optimal/suboptimal solution for the original RAB problem by solving the BLMI formulations. To validate our results, simulation examples are presented to demonstrate the improved array output SINR of the proposed robust beamformer.

Stochastic Traffic assignment with Electric Vehicles: a convex optimization approach

Electric vehicles and charging stations are increasing worldwide to reduce emissions at the city level. In order to analyze traffic flows and quantify the demand for energy recharge, it is necessary to model users’ choices considering different kinds of traffic flows. In this paper, attention is focused on the formalization of the joint traffic and energy demand assignment conditions over a traffic network in presence of electric vehicles (EVs). Specifically, the so-called stochastic user equilibrium (SUE) conditions are used to define the steady state behaviour of a generic traffic network in which some arcs are equipped with EV charging stations. A convex optimization problem is defined and solved, which is equivalent to the solution of the nonlinear and nonconvex set of SUE conditions under a multinomial Logit user's choice function. A real case study, related to a touristic area in the Liguria Region (from Genova airport to Portofino), is considered.

AEGD: adaptive gradient descent with energy

We propose AEGD, a new algorithm for optimization of non-convex objective functions, based on a dynamically updated 'energy' variable. The method is shown to be unconditionally energy stable, irrespective of the base step size. We prove energy-dependent convergence rates of AEGD for both non-convex and convex objectives, which for a suitably small step size recovers desired convergence rates for the batch gradient descent. We also provide an energy-dependent bound on the stationary convergence of AEGD in the stochastic non-convex setting. The method is straightforward to implement and requires little tuning of hyper-parameters. Experimental results demonstrate that AEGD works well for a large variety of optimization problems. Specifically, it is robust with respect to initial data, capable of making rapid initial progress. The stochastic AEGD shows comparable and often better generalization performance than SGD with momentum for deep neural networks. The code is available at https://github.com/txping/AEGD.

Comparison theorems for evolution inclusions with maximal monotone operators. $L^2$-theory

An evolution inclusion with time-dependent family of maximal monotone operators in considered in a separable Hilbert space. If the elements with minimum norm of the family of maximal monotone operators satisfy certain growth conditions, then the domains of definition of this family are closed convex sets. Hence the sweeping process is well defined, whose values are the normal cones of the domains of definition of maximal monotone operators. It is shown that if the sweeping process has a solution for each single-valued perturbation from the space of integrable functions, then the evolution inclusion with the maximal monotone operators and single-valued perturbations from the space of integrable functions is also solvable. Quite general conditions in terms of the properties of the family of maximal monotone operators that ensure the existence of solutions for the sweeping process are presented. All results obtained and the approach presented are new. They are used to prove an existence theorem for evolution inclusions with multivalued perturbations, whose values are closed nonconvex sets. Bibliography: 19 titles.

Generalized $ (f, \lambda) $-projection operator on closed nonconvex sets and its applications in reflexive smooth Banach spaces

<abstract><p>In this paper, we expanded from the convex case to the nonconvex case in the setting of reflexive smooth Banach spaces, the concept of the $ f $-generalized projection $ \pi^{f}_S:X^*\to S $ initially introduced for convex sets and convex functions in <sup>[<xref ref-type="bibr" rid="b19">19</xref>,<xref ref-type="bibr" rid="b20">20</xref>]</sup>. Indeed, we defined the $ (f, \lambda) $-generalized projection operator $ \pi^{f, \lambda}_S:X^*\to S $ from $ X^* $ onto a nonempty closed set $ S $. We proved many properties of $ \pi^{f, \lambda}_S $ for any closed (not necessarily convex) set $ S $ and for any lower semicontinuous function $ f $. Our principal results broaden the scope of numerous theorems established in <sup>[<xref ref-type="bibr" rid="b19">19</xref>,<xref ref-type="bibr" rid="b20">20</xref>]</sup> from the convex setting to the nonconvex setting. An application of our main results to solutions of nonconvex variational problems is stated at the end of the paper.</p></abstract>

Stochastic Zeroth-Order Functional Constrained Optimization: Oracle Complexity and Applications

Functionally constrained stochastic optimization problems, where neither the objective function nor the constraint functions are analytically available, arise frequently in machine learning applications. In this work, assuming we only have access to the noisy evaluations of the objective and constraint functions, we propose and analyze stochastic zeroth-order algorithms for solving this class of stochastic optimization problem. When the domain of the functions is [Formula: see text], assuming there are m constraint functions, we establish oracle complexities of order [Formula: see text] and [Formula: see text] in the convex and nonconvex settings, respectively, where ϵ represents the accuracy of the solutions required in appropriately defined metrics. The established oracle complexities are, to our knowledge, the first such results in the literature for functionally constrained stochastic zeroth-order optimization problems. We demonstrate the applicability of our algorithms by illustrating their superior performance on the problem of hyperparameter tuning for sampling algorithms and neural network training.Funding: K. Balasubramanian was partially supported by a seed grant from the Center for Data Science and Artificial Intelligence Research, University of California–Davis, and the National Science Foundation [Grant DMS-2053918].

Curvature measures and soap bubbles beyond convexity

Extending the celebrated results of Alexandrov (1958) and Korevaar–Ros (1988) for smooth sets, as well as the results of Schneider (1979) and the first author (1999) for arbitrary convex bodies, we obtain for the first time the characterization of the isoperimetric sets of a uniformly convex smooth finite-dimensional normed space (i.e. Wulff shapes) in the non-smooth and non-convex setting, based on a natural geometric condition involving the curvature measures. More specifically we show, under a weak mean convexity assumption, that finite unions of disjoint Wulff shapes are the only sets of positive reach A⊆Rn+1 with finite and positive volume such that, for some k∈{0,…,n−1}, the k-th generalized curvature measure Θkϕ(A,⋅), which is defined on the unit normal bundle of A with respect to the relative geometry induced by ϕ, is proportional to the top order generalized curvature measure Θnϕ(A,⋅). If k=n−1 the conclusion holds for all sets of positive reach with finite and positive volume. We also prove a related sharp result about the removability of the singularities. This result is based on the extension of the notion of a normal boundary point, originally introduced by Busemann and Feller (1936) for arbitrary convex bodies, to sets of positive reach.These findings are new even in the Euclidean space.Several auxiliary and related results are proved, which are of independent interest. They include the extension of the classical Steiner–Weyl tube formula to arbitrary closed sets in a finite dimensional uniformly convex normed vector space, a general formula for the derivative of the localized volume function, which extends and complements recent results of Chambolle–Lussardi–Villa (2021), and general versions of the Heintze–Karcher inequality.

Adaptive Stochastic Gradient Descent Method for Convex and Non-Convex Optimization

Stochastic gradient descent is the method of choice for solving large-scale optimization problems in machine learning. However, the question of how to effectively select the step-sizes in stochastic gradient descent methods is challenging, and can greatly influence the performance of stochastic gradient descent algorithms. In this paper, we propose a class of faster adaptive gradient descent methods, named AdaSGD, for solving both the convex and non-convex optimization problems. The novelty of this method is that it uses a new adaptive step size that depends on the expectation of the past stochastic gradient and its second moment, which makes it efficient and scalable for big data and high parameter dimensions. We show theoretically that the proposed AdaSGD algorithm has a convergence rate of O(1/T) in both convex and non-convex settings, where T is the maximum number of iterations. In addition, we extend the proposed AdaSGD to the case of momentum and obtain the same convergence rate for AdaSGD with momentum. To illustrate our theoretical results, several numerical experiments for solving problems arising in machine learning are made to verify the promise of the proposed method.

A nonconvex function activated noise-tolerant neurodynamic model aided with Fischer-Burmeister function for time-varying quadratic programming in the presence of noises

Time-varying quadratic programming problems (TVQPPs) with equality and inequality constraints often arise in the fields of scientific computation and engineering application. Zeroing neural network (ZNN), being a special kind of recurrent neural network, has shown powerful capabilities to compute a variety of zeroing finding problems with monotonically increasing odd activation function. However, the projection sets of nonconvex activation function are obviously excluded, which means a general conclusion remain unexplored. In addition, noises are always ubiquitous in actual applications. Nevertheless, most existing ZNN-based models usually assume that the solving process is free of noise before the calculation. In this paper, a general zeroing neural network model with nonconvex activated function (GZNNM-NAF) and a general noise-tolerant zeroing neural network model with nonconvex activated function (GNTZNNM-NAF), which are also viewed as ZNN-type models, are developed by the inspiration of the traditional ZNN model from a control-based perspective. The ZNN-type models break the limitation of the traditional ZNN models of activation function, which allows nonconvex sets for projection operations and combines nonlinear complementary function for dealing with inequality constraints arising in TVQPPs. Moreover, theoretical results indicate that the ZNN-type models globally converge to time-varying optimal solution of TVQPPs with equality and inequality constraints under the noise circumstance. According to the different cases of nonconvex activation function, robustness analyses are demonstrated in detail for TVQPPs. It may enlarge the scope of the proposed method, especially in the field of practical application. Finally, a numerical example and an application example to manipulator motion generation are analyzed to verify the superiority and robustness of the developed ZNN-type models for TVQPPs with different measurement noises.

Data-driven robust optimization using deep neural networks

Robust optimization has been established as a leading methodology to approach decision problems under uncertainty. To derive a robust optimization model, a central ingredient is to identify a suitable model for uncertainty, which is called the uncertainty set. An ongoing challenge in the recent literature is to derive uncertainty sets from given historical data that result in solutions that are robust regarding future scenarios. In this paper we use an unsupervised deep learning method to learn and extract hidden structures and anomalies from data, leading to non-convex uncertainty sets and better robust solutions. We prove that most of the classical uncertainty classes are special cases of our derived sets and that optimizing over them is strongly NP-hard. Nevertheless, we show that the trained neural networks can be integrated into a robust optimization model by formulating the adversarial problem as a convex quadratic mixed-integer program. This allows us to derive robust solutions through an iterative scenario generation process. In our computational experiments, we compare this approach to a similar approach using kernel-based support vector clustering and to other benchmark methods. We find that uncertainty sets derived by the unsupervised deep learning method find a better description of data and lead to robust solutions that often outperform the comparison methods both with respect to objective value and feasibility.

Global linear convergence of evolution strategies with recombination on scaling-invariant functions

Evolution Strategies (ESs) are stochastic derivative-free optimization algorithms whose most prominent representative, the CMA-ES algorithm, is widely used to solve difficult numerical optimization problems. We provide the first rigorous investigation of the linear convergence of step-size adaptive ESs involving a population and recombination, two ingredients crucially important in practice to be robust to local irregularities or multimodality. We investigate the convergence of step-size adaptive ESs with weighted recombination on composites of strictly increasing functions with continuously differentiable scaling-invariant functions with a global optimum. This function class includes functions with non-convex sublevel sets and discontinuous functions. We prove the existence of a constant r such that the logarithm of the distance to the optimum divided by the number of iterations converges to r. The constant is given as an expectation with respect to the stationary distribution of a Markov chain—its sign allows to infer linear convergence or divergence of the ES and is found numerically. Our main condition for convergence is the increase of the expected log step-size on linear functions. In contrast to previous results, our condition is equivalent to the almost sure geometric divergence of the step-size on linear functions.

A new projective exact penalty function for a general constrained optimization

A new projective exact penalty function method is proposed for the equivalent reduction of constrained optimization problems to unconstrained ones. In the method, the original objective function is extended to infeasible points by summing its value at the projection of an infeasible point on the feasible set with the distance to the set. The equivalence means that local and global minimums of the problems coincide. Nonconvex sets with multivalued projections are admitted, and the objective function may be lower semicontinuous. The particular case of convex problems is included. So the method does not assume the existence of the objective function outside the allowable area and does not require the selection of the penalty coefficient.

A Sharper Computational Tool for Regression

Building on previous research of Chi and Chi, this article revisits estimation in robust structured regression under the criterion. We adopt the majorization-minimization (MM) principle to design a new algorithm for updating the vector of regression coefficients. Our sharp majorization achieves faster convergence than the previous alternating proximal gradient descent algorithm by Chi and Chi. In addition, we reparameterize the model by substituting precision for scale and estimate precision via a modified Newton’s method. This simplifies and accelerates overall estimation. We also introduce distance-to-set penalties to enable constrained estimation under nonconvex constraint sets. This tactic also improves performance in coefficient estimation and structure recovery. Finally, we demonstrate the merits of our improved tactics through a rich set of simulation examples and a real data application.

An augmented Lagrangian method for optimization problems with structured geometric constraints

This paper is devoted to the theoretical and numerical investigation of an augmented Lagrangian method for the solution of optimization problems with geometric constraints. Specifically, we study situations where parts of the constraints are nonconvex and possibly complicated, but allow for a fast computation of projections onto this nonconvex set. Typical problem classes which satisfy this requirement are optimization problems with disjunctive constraints (like complementarity or cardinality constraints) as well as optimization problems over sets of matrices which have to satisfy additional rank constraints. The key idea behind our method is to keep these complicated constraints explicitly in the constraints and to penalize only the remaining constraints by an augmented Lagrangian function. The resulting subproblems are then solved with the aid of a problem-tailored nonmonotone projected gradient method. The corresponding convergence theory allows for an inexact solution of these subproblems. Nevertheless, the overall algorithm computes so-called Mordukhovich-stationary points of the original problem under a mild asymptotic regularity condition, which is generally weaker than most of the respective available problem-tailored constraint qualifications. Extensive numerical experiments addressing complementarity- and cardinality-constrained optimization problems as well as a semidefinite reformulation of MAXCUT problems visualize the power of our approach.

An Adaptive Alternating Direction Method of Multipliers

The alternating direction method of multipliers (ADMM) is a powerful splitting algorithm for linearly constrained convex optimization problems. In view of its popularity and applicability, a growing attention is drawn toward the ADMM in nonconvex settings. Recent studies of minimization problems for nonconvex functions include various combinations of assumptions on the objective function including, in particular, a Lipschitz gradient assumption. We consider the case where the objective is the sum of a strongly convex function and a weakly convex function. To this end, we present and study an adaptive version of the ADMM which incorporates generalized notions of convexity and penalty parameters adapted to the convexity constants of the functions. We prove convergence of the scheme under natural assumptions. To this end, we employ the recent adaptive Douglas–Rachford algorithm by revisiting the well-known duality relation between the classical ADMM and the Douglas–Rachford splitting algorithm, generalizing this connection to our setting. We illustrate our approach by relating and comparing to alternatives, and by numerical experiments on a signal denoising problem.

Gradient dynamics in reinforcement learning.

Despite the success achieved by the analysis of supervised learning algorithms in the framework of statistical mechanics, reinforcement learning has remained largely untouched by physicists. Here we move towards closing the gap by analyzing the dynamics of the policy gradient algorithm. For a convex problem, namely the k-armed bandit, we show that the learning dynamics obeys a drift-diffusion motion described by a Langevin equation, the coefficients of which can be tuned by the learning rate. We explore the striking similarity between our Langevin equationand the Kimura equation, describing genotypes evolution. Furthermore, we propose a mapping between a nonconvex reinforcement learning setting describing multiple joints of a robotic arm and a disordered system, namely a p-spin glass. This mapping enables us to show how the learning rate acts as an effective temperature and thus is capable of smoothing rough landscapes, corroborating what is displayed by the drift-diffusive description and paving the way for physics-inspired algorithmic optimization based on annealing procedures in disordered systems.

Inertial alternating direction method of multipliers for non-convex non-smooth optimization

In this paper, we propose an algorithmic framework, dubbed inertial alternating direction methods of multipliers (iADMM), for solving a class of nonconvex nonsmooth multiblock composite optimization problems with linear constraints. Our framework employs the general minimization-majorization (MM) principle to update each block of variables so as to not only unify the convergence analysis of previous ADMM that use specific surrogate functions in the MM step, but also lead to new efficient ADMM schemes. To the best of our knowledge, in the nonconvex nonsmooth setting, ADMM used in combination with the MM principle to update each block of variables, and ADMM combined with \emph{inertial terms for the primal variables} have not been studied in the literature. Under standard assumptions, we prove the subsequential convergence and global convergence for the generated sequence of iterates. We illustrate the effectiveness of iADMM on a class of nonconvex low-rank representation problems.