Nonconvex Case Research Articles

A new view on risk measures associated with acceptance sets

In this paper, we study properties of certain risk measures associated with acceptance sets. These sets describe regulatory preconditions that have to be fulfilled by financial institutions to pass a given acceptance test. If the financial position of an institution is not acceptable, the decision maker has to raise new capital and invest it into a basket of so called eligible assets to change the current position such that the resulting one corresponds with an element of the acceptance set. Risk measures have been widely studied. The risk measure that is considered here determines the minimal costs of making a financial position acceptable. In the literature, monetary risk measures are often defined as translation invariant functions and, thus, there is an equivalent formulation as Gerstewitz-Functional. The Gerstewitz-Functional is an useful tool for separation and scalarization in multiobjective optimization in the non-convex case. In our paper, we study properties of the sublevel sets, strict sublevel sets and level lines of a risk measure defined on a linear space. Furthermore, we discuss the finiteness of the risk measure and relax the closedness assumptions.

Communication-Efficient Distributed SGD with Error-Feedback, Revisited

We show that the convergence proof of a recent algorithm called dist-EF-SGD for distributed stochastic gradient descent with communication efficiency using error-feedback of Zheng et al. (NeurIPS 2019) is problematic mathematically. Concretely, the original error bound for arbitrary sequences of learning rate is unfortunately incorrect, leading to an invalidated upper bound in the convergence theorem for the algorithm. As evidences, we explicitly provide several counter-examples, for both convex and non-convex cases, to show the incorrectness of the error bound. We fix the issue by providing a new error bound and its corresponding proof, leading to a new convergence theorem for the dist-EF-SGD algorithm, and therefore recovering its mathematical analysis.

Dynamic string-averaging CQ-methods for the split feasibility problem with percentage violation constraints arising in radiation therapy treatment planning.

We study a feasibility-seeking problem with percentage violation constraints (PVCs). These are additional constraints that are appended to an existing family of constraints, which single out certain subsets of the existing constraints and declare that up to a specified fraction of the number of constraints in each subset is allowed to be violated by up to a specified percentage of the existing bounds. Our motivation to investigate problems with PVCs comes from the field of radiation therapy treatment planning (RTTP) wherein the fully discretized inverse planning problem is formulated as a split feasibility problem and the PVCs give rise to nonconvex constraints. Following the CQ algorithm of Byrne (2002, Inverse Problems, Vol. 18, pp. 441-53), we develop a string-averaging CQ-method that uses only projections onto the individual sets that are half-spaces represented by linear inequalities. The question of extending our theoretical results to the nonconvex sets case is still open. We describe how our results apply to RTTP and provide a numericalexample.

Intermittent Pulling with Local Compensation for Communication-Efficient Distributed Learning

As a widely used iterative algorithm, the distributed Stochastic Gradient Descent (SGD) has shown great advances in training machine learning models due to the reduced time of the gradients computation. However, the huge number of iterations of SGD usually incurs huge communication cost on pushing local gradients and pulling global model that prohibits its further improvement over performance. In this paper, to reduce the number of pulling operations, a novel approach named Pulling Reduction with Local Compensation (PRLC) is proposed, in which each worker intermittently pulls the global model from the server and uses its local update to compensate the gap between the local model and the global model. Our rigorous theoretical analysis shows that the convergence rate of PRLC preserves the same order as the classical synchronous SGD for both strongly-convex and non-convex cases with good scalability due to the linear speedup with respect to the number of training nodes. Moreover, we also show that PRLC admits lower pulling frequency than the pulling reduction method without local compensation. The extensive experiments conducted on various models exhibit that our approach achieves a significant pulling reduction over the state-of-the-art methods, e.g., requiring only half of the pulling operations of LAG.

Unsupervised feature selection for attributed graphs

Many real-world applications generate attributed graphs that contain both link structures and content information associated with nodes. Content information in real networks always contains high dimensional feature space. In recent years, unsupervised feature selection has been widely used in handling high dimensional data without label information. Most existing unsupervised feature selection methods assume that instances in datasets are independent and identically distributed. However, instances in attributed graphs are intrinsically correlated. Considering the wide applications of feature selection in attributed graphs, we propose a new unsupervised feature selection method based on regularized sparse learning. We use pseudo class labels to learn the interdependency from both link and content information, and embed the obtained information into a sparse learning based feature selection framework. In particular, a new regularization term is designed to learn link information, which capture group behavior among the connected instances utilizing latent social dimensions. To solve the proposed feature selection model, we consider both convex and nonconvex cases and design the corresponding algorithms based on the Alternating Direction Method of Multipliers (ADMM) combined with ConCave Convex Procedure (CCCP). Numerical studies are implemented on real-world datasets to validate the advantage of our new method.

Nonconvex Equilibrium Problems Via a KKM Theorem

In this paper, we establish existence results for vector equilibrium problems without convexity assumptions on the whole domain, but just on the closure of a quasi-self-segment-dense subset. In the absence of demanding technical assumptions, we present new conditions to obtain results on the existence of solutions of nonconvex equilibrium problems in real topological vector spaces via a KKM theorem. Furthermore, no coercivity conditions are used to deal with the noncompact case in reflexive Banach spaces. Examples are provided to compare the obtained results with the literature in convex and nonconvex cases.

New convergence results for the inexact variable metric forward–backward method

Forward–backward methods are valid tools to solve a variety of optimization problems where the objective function is the sum of a smooth, possibly nonconvex term plus a convex, possibly nonsmooth function. The corresponding iteration is built on two main ingredients: the computation of the gradient of the smooth part and the evaluation of the proximity (or resolvent) operator associated with the convex term. One of the main difficulties, from both implementation and theoretical point of view, arises when the proximity operator is computed in an inexact way. The aim of this paper is to provide new convergence results about forward–backward methods with inexact computation of the proximity operator, under the assumption that the objective function satisfies the Kurdyka–Łojasiewicz property. In particular, we adopt an inexactness criterion which can be implemented in practice, while preserving the main theoretical properties of the proximity operator. The main result is the proof of the convergence of the iterates generated by the forward–backward algorithm in Bonettini et al. (2017) to a stationary point. Convergence rate estimates are also provided. At the best of our knowledge, there exists no other inexact forward–backward algorithm with proved convergence in the nonconvex case and equipped with an explicit procedure to inexactly compute the proximity operator.

Fractional-Order Integro-Differential Multivalued Problems with Fixed and Nonlocal Anti-Periodic Boundary Conditions

This paper studies a new class of fractional differential inclusions involving two Caputo fractional derivatives of different orders and a Riemann–Liouville type integral nonlinearity, supplemented with a combination of fixed and nonlocal (dual) anti-periodic boundary conditions. The existence results for the given problem are obtained for convex and non-convex cases of the multi-valued map by applying the standard tools of the fixed point theory. Examples illustrating the obtained results are presented.

On the Extension of the DIRECT Algorithm to Multiple Objectives

Deterministic global optimization algorithms like Piyavskii–Shubert, direct, ego and many more, have a recognized standing, for problems with many local optima. Although many single objective optimization algorithms have been extended to multiple objectives, completely deterministic algorithms for nonlinear problems with guarantees of convergence to global Pareto optimality are still missing. For instance, deterministic algorithms usually make use of some form of scalarization, which may lead to incomplete representations of the Pareto optimal set. Thus, all global Pareto optima may not be obtained, especially in nonconvex cases. On the other hand, algorithms attempting to produce representations of the globally Pareto optimal set are usually based on heuristics. We analyze the concept of global convergence for multiobjective optimization algorithms and propose a convergence criterion based on the Hausdorff distance in the decision space. Under this light, we consider the well-known global optimization algorithm direct, analyze the available algorithms in the literature that extend direct to multiple objectives and discuss possible alternatives. In particular, we propose a novel definition for the notion of potential Pareto optimality extending the notion of potential optimality defined in direct. We also discuss its advantages and disadvantages when compared with algorithms existing in the literature.

The Gradient Projection Algorithm for Smooth Sets and Functions in Nonconvex Case

We consider the problem of minimization for a function with Lipschitz continuous gradient on a proximally smooth and smooth manifold in a finite dimensional Euclidean space. We consider the Lezanski-Polyak-Lojasiewicz (LPL) conditions in this problem of constrained optimization. We prove that the gradient projection algorithm for the problem converges with a linear rate when the LPL condition holds.

Recession function and its applications in optimization

ABSTRACT In this paper, we focus on the recession cone and recession function in the nonconvex case. By virtue of the recession function, we investigate the monotonicity of a function. Then, we obtain a characterization of the Lipschitz continuity for a function by using the recession function. As applications, we study the monotonicity and Lipschitz continuity properties of the optimal value function for a parametric constrained optimization problem. Moreover, we present a characterization of the nonemptiness and boundedness of the solution set for a constrained optimization problem without convexity assumption.

Proximal methods for reweighted l-regularization of sparse signal recovery

Abstract To recover a sparse signal from a noised linear measurement system A x = b + e , convex lp regularization methods (i.e., 1 ≤ p p = 1 ) are commonly used under certain conditions. Recently, however, more attentions have been paid to nonconvex lq regularization methods (i.e., 0 q = 1 / 2 ) for recovering a sparse signal. In this paper, we use proximal methods to study both convex and nonconvex reweighted lQ regularization for recovering a sparse signal. Convex lQ regularization is introduced by S. Voronin and I. Daubechies [19]. We extend it to the nonconvex case and our results therefore supplement those of Voronin and Daubechies [19] . We also study Nesterov’s acceleration method for the nonconvex case. Our numerical experiments show that nonconvex lQ regularization can more effectively recover sparse signals.

NETT: solving inverse problems with deep neural networks

Recovering a function or high-dimensional parameter vector from indirect measurements is a central task in various scientific areas. Several methods for solving such inverse problems are well developed and well understood. Recently, novel algorithms using deep learning and neural networks for inverse problems appeared. While still in their infancy, these techniques show astonishing performance for applications like low-dose CT or various sparse data problems. However, there are few theoretical results for deep learning in inverse problems. In this paper, we establish a complete convergence analysis for the proposed NETT (network Tikhonov) approach to inverse problems. NETT considers nearly data-consistent solutions having small value of a regularizer defined by a trained neural network. We derive well-posedness results and quantitative error estimates, and propose a possible strategy for training the regularizer. Our theoretical results and framework are different from any previous work using neural networks for solving inverse problems. A possible data driven regularizer is proposed. Numerical results are presented for a tomographic sparse data problem, which demonstrate good performance of NETT even for unknowns of different type from the training data. To derive the convergence and convergence rates results we introduce a new framework based on the absolute Bregman distance generalizing the standard Bregman distance from the convex to the non-convex case.

Three-Point Boundary Value Problems for the Langevin Equation with the Hilfer Fractional Derivative

We discuss the existence and uniqueness of solutions for the Langevin fractional differential equation and its inclusion counterpart involving the Hilfer fractional derivatives, supplemented with three-point boundary conditions by means of standard tools of the fixed-point theorems for single and multivalued functions. We make use of Banach’s fixed-point theorem to obtain the uniqueness result, while the nonlinear alternative of the Leray-Schauder type and Krasnoselskii’s fixed-point theorem are applied to obtain the existence results for the single-valued problem. Existence results for the convex and nonconvex valued cases of the inclusion problem are derived via the nonlinear alternative for Kakutani’s maps and Covitz and Nadler’s fixed-point theorem respectively. Examples illustrating the obtained results are also constructed. (2010) Mathematics Subject Classifications. This study is classified under the following classification codes: 26A33; 34A08; 34A60; and 34B15.

ApproxIt: A Quality Management Framework of Approximate Computing for Iterative Methods

Approximate computing, being able to tradeoff computation quality (e.g., accuracy) and computational effort (e.g., energy) for error-tolerant applications such as media processing and the emerging recognition, mining, and synthesis (RMS) applications, has gained significant traction in recent years. With approximate computing, we expect to obtain acceptable results, but how do we make sure the quality of the final results are good enough? This challenging problems remains largely unexplored. As many of the RMS applications employ iterative methods (IMs) for solution-finding, wherein a sequence of improving approximate solutions are generated before reaching the final converged solution, in this paper, we propose ApproxIt , a novel quality management framework of approximate computing dedicated for IMs with quality guarantees. ApproxIt is comprised of two stages: 1) offline stage and 2) online stage. To be specific, at offline stage, we first analyze the manifold of parameter space to identify the given problem as convex case or nonconvex case at the offline stage. And for each case, we propose the corresponding runtime dynamic quality calibration scheme and reconfiguration control policy. Then during runtime, our proposed lightweight quality estimator will evaluate the intermediate quality at specific calibration iteration, which is determined by the novel Markov model-based calibration scheme. If quality violation occurs, the configuration control policy will select the most appropriate approximate computing mode for the following iterations. With the proposed dynamic effort scaling technique, ApproxIt is able to dramatically improve application energy efficiency under quality guarantees, as demonstrated in our experimental results.

A robust graph-based semi-supervised sparse feature selection method

Feature selection is used for excluding redundant features and enhancing learning performance. Abundant unlabeled data are existed in many applications which can be used in semi-supervised feature selection. Semi-supervised sparse feature selection methods have been presented to apply labeled and unlabeled data and consider the correlation among features. Most of the sparse methods utilize square-norm based loss function which is sensitive to outliers. In this paper, we propose a robust Graph-based Semi-Supervised Sparse Feature Selection (GS3FS) method based on the mixed convex and non-convex l2,p-norm (0 < p ≤ 1) minimization on regularization and loss function. The l2,p-norm based regularization ensures that the method selects relevant and sparse features, and the l2,p-norm based loss function makes the method robust to outliers. Proposed method takes the advantages of manifold learning and l2,p-norm minimization on regularization and loss function. The method includes the mixed l2,p-norm which is difficult to solve. To solve the proposed method for the convex (p = 1) and non-convex (0 < p <1) cases, we propose an efficient unified algorithm and prove its convergence. Extensive experiments are carried out on various classification and regression datasets for evaluating the proposed method. The results indicate that the proposed method selects the relevant features and improves the performance of classification and regression models.

Improving Lagrange Dual Bounds for Quadratic Extremal Problems

Introduction. Due to the fact that quadratic extremal problems are generally NP-hard, various convex relaxations to find bounds for their global extrema are used, namely, Lagrangian relaxation, SDP-relaxation, SOCP-relaxation, LP-relaxation, and others. This article investigates a dual bound that results from the Lagrangian relaxation of all constraints of quadratic extremal problem. The main issue when using this approach for solving quadratic extremal problems is the quality of the obtained bounds (the magnitude of the duality gap) and the possibility to improve them. While for quadratic convex optimization problems such bounds are exact, in other cases this issue is rather complicated. In non-convex cases, to improve the dual bounds (to reduce the duality gap) the techniques, based on ambiguity of the problem formulation, can be used. The most common of these techniques is an extension of the original quadratic formulation of the problem by introducing the so-called functionally superfluous constraints (additional constraints that result from available constraints). The ways to construct such constraints can be general in nature or they can use specific features of the concrete problems. The purpose of the article is to propose methods for improving the Lagrange dual bounds for quadratic extremal problems by using technique of functionally superfluous constraints; to present examples of constructing such constraints. Results. The general concept of using functionally superfluous constraints for improving the Lagrange dual bounds for quadratic extremal problems is considered. Methods of constructing such constraints are presented. In particular, the method proposed by N.Z. Shor for constructing functionally superfluous constraints for quadratic problems of general form is presented in generalized and schematized forms. Also it is pointed out that other special techniques, which employ the features of specific problems for constructing functionally superfluous constraints, can be used. Conclusions. In order to improve dual bounds for quadratic extremal problems, one can use various families of functionally superfluous constraints, both of general and specific type. In some cases, their application can improve bounds or even provide an opportunity to obtain exact values of global extrema.

Sparse feature selection in multi-target modeling of carbonic anhydrase isoforms by exploiting shared information among multiple targets

Multi-target modeling can be used for inhibition prediction of CA isoforms, the essential zinc metalloenzymes involved in different biological processes such as tumorigenesis. In this study, the first multi-target model is developed for predicting the activities of inhibitors against CA-I, CA-II, CA-IX, and CA-XII. Structural similarity analysis is carried out for two cancer-related isoforms CA-IX and CA-XII. The mean TM-score value (0.935) reveals a marked similarity between the two structures. To select relevant descriptors for the developed multi-target model, we propose a novel feature selection method based on shared subspace learning, which considers correlation among different targets in multi-target modeling. The proposed shared subspace feature selection method uses the mixed convex and non-convex l2,p-norm (0 ​< ​p ​≤ ​1) minimization on both regularization and loss function to ensure that the loss function is robust to outliers and consider correlation among different descriptors for joint sparse feature selection. To solve the proposed shared subspace feature selection method for convex and non-convex cases, a unified Algorithm is presented. The study utilized a test set to evaluate the performance of the proposed feature selection method with a multi-target kernel smoother model and compare to that of other feature selection methods with the multi-target kernel smoother models. The obtained results demonstrate the superiority of the proposed shared-subspace feature selection approach based on l2,1/2-norm in selecting the most relevant descriptors. Statistical results (RMSEtest ​= ​0.5190, R2test ​= ​0.7613 and Q2ext ​= ​0.7524) demonstrate that the model displays adequate quality for virtual screening. The results also represent the significance of using the shared subspace among different targets in the selection of relevant descriptors to predict the inhibition of CA isoforms.

Global Convergence of the (1 + 1) Evolution Strategy to a Critical Point.

We establish global convergence of the (1 + 1) evolution strategy, that is, convergence to a critical point independent of the initial state. More precisely, we show the existence of a critical limit point, using a suitable extension of the notion of a critical point to measurable functions. At its core, the analysis is based on a novel progress guarantee for elitist, rank-based evolutionary algorithms. By applying it to the (1 + 1) evolution strategy we are able to provide an accurate characterization of whether global convergence is guaranteed with full probability, or whether premature convergence is possible. We illustrate our results on a number of example applications ranging from smooth (non-convex) cases over different types of saddle points and ridge functions to discontinuous and extremely rugged problems.

Scenario Approach for Minmax Optimization with Emphasis on the Nonconvex Case: Positive Results and Caveats

We treat the so-called scenario approach, a popular probabilistic approximation method for robust minmax optimization problems via independent and identically distributed (i.i.d.) sampling from the...