Convex Case Research Articles

Global Convergence Guarantees of (A)GIST for a Family of Nonconvex Sparse Learning Problems.

In recent years, most of the studies have shown that the generalized iterated shrinkage thresholdings (GISTs) have become the commonly used first-order optimization algorithms in sparse learning problems. The nonconvex relaxations of the l0 -norm usually achieve better performance than the convex case (e.g., l1 -norm) since the former can achieve a nearly unbiased solver. To increase the calculation efficiency, this work further provides an accelerated GIST version, that is, AGIST, through the extrapolation-based acceleration technique, which can contribute to reduce the number of iterations when solving a family of nonconvex sparse learning problems. Besides, we present the algorithmic analysis, including both local and global convergence guarantees, as well as other intermediate results for the GIST and AGIST, denoted as (A)GIST, by virtue of the Kurdyka-Łojasiewica (KŁ) property and some milder assumptions. Numerical experiments on both synthetic data and real-world databases can demonstrate that the convergence results of objective function accord to the theoretical properties and nonconvex sparse learning methods can achieve superior performance over some convex ones.

Inexact Proximal Point Methods for Multiobjective Quasiconvex Minimization on Hadamard Manifolds

In this paper, we present two inexact scalarization proximal point methods to solve quasiconvex multiobjective minimization problems on Hadamard manifolds. Under standard assumptions on the problem, we prove that the two sequences generated by the algorithms converge to a Pareto critical point of the problem and, for the convex case, the sequences converge to a weak Pareto solution. Finally, we explore an application of the method to demand theory in economy, which can be dealt with using the proposed algorithm.

Stochastic Gradient Descent in Continuous Time: A Central Limit Theorem

Stochastic gradient descent in continuous time (SGDCT) provides a computationally efficient method for the statistical learning of continuous-time models, which are widely used in science, engineering, and finance. The SGDCT algorithm follows a (noisy) descent direction along a continuous stream of data. The parameter updates occur in continuous time and satisfy a stochastic differential equation. This paper analyzes the asymptotic convergence rate of the SGDCT algorithm by proving a central limit theorem for strongly convex objective functions and, under slightly stronger conditions, for nonconvex objective functions as well. An [Formula: see text] convergence rate is also proven for the algorithm in the strongly convex case. The mathematical analysis lies at the intersection of stochastic analysis and statistical learning.

Boundary value problems for Caputo-Hadamard fractional differential inclusions with Integral Conditions

Abstract For r ∈ (1, 2], the authors establish sufficient conditions for the existence of solutions for a class of boundary value problem for rth order Caputo-Hadamard fractional differential inclusions satisfying nonlinear integral conditions. Both cases of convex and nonconvex valued right hand sides are considered.

Randomized sketch descent methods for non-separable linearly constrained optimization

Abstract In this paper we consider large-scale smooth optimization problems with multiple linear coupled constraints. Due to the non-separability of the constraints, arbitrary random sketching would not be guaranteed to work. Thus, we first investigate necessary and sufficient conditions for the sketch sampling to have well-defined algorithms. Based on these sampling conditions we develop new sketch descent methods for solving general smooth linearly constrained problems, in particular, random sketch descent (RSD) and accelerated random sketch descent (A-RSD) methods. To our knowledge, this is the first convergence analysis of RSD algorithms for optimization problems with multiple non-separable linear constraints. For the general case, when the objective function is smooth and non-convex, we prove for the non-accelerated variant sublinear rate in expectation for an appropriate optimality measure. In the smooth convex case, we derive for both algorithms, non-accelerated and A-RSD, sublinear convergence rates in the expected values of the objective function. Additionally, if the objective function satisfies a strong convexity type condition, both algorithms converge linearly in expectation. In special cases, where complexity bounds are known for some particular sketching algorithms, such as coordinate descent methods for optimization problems with a single linear coupled constraint, our theory recovers the best known bounds. Finally, we present several numerical examples to illustrate the performances of our new algorithms.

Three-Point Boundary Value Problems for the Langevin Equation with the Hilfer Fractional Derivative

We discuss the existence and uniqueness of solutions for the Langevin fractional differential equation and its inclusion counterpart involving the Hilfer fractional derivatives, supplemented with three-point boundary conditions by means of standard tools of the fixed-point theorems for single and multivalued functions. We make use of Banach’s fixed-point theorem to obtain the uniqueness result, while the nonlinear alternative of the Leray-Schauder type and Krasnoselskii’s fixed-point theorem are applied to obtain the existence results for the single-valued problem. Existence results for the convex and nonconvex valued cases of the inclusion problem are derived via the nonlinear alternative for Kakutani’s maps and Covitz and Nadler’s fixed-point theorem respectively. Examples illustrating the obtained results are also constructed. (2010) Mathematics Subject Classifications. This study is classified under the following classification codes: 26A33; 34A08; 34A60; and 34B15.

ApproxIt: A Quality Management Framework of Approximate Computing for Iterative Methods

Approximate computing, being able to tradeoff computation quality (e.g., accuracy) and computational effort (e.g., energy) for error-tolerant applications such as media processing and the emerging recognition, mining, and synthesis (RMS) applications, has gained significant traction in recent years. With approximate computing, we expect to obtain acceptable results, but how do we make sure the quality of the final results are good enough? This challenging problems remains largely unexplored. As many of the RMS applications employ iterative methods (IMs) for solution-finding, wherein a sequence of improving approximate solutions are generated before reaching the final converged solution, in this paper, we propose ApproxIt , a novel quality management framework of approximate computing dedicated for IMs with quality guarantees. ApproxIt is comprised of two stages: 1) offline stage and 2) online stage. To be specific, at offline stage, we first analyze the manifold of parameter space to identify the given problem as convex case or nonconvex case at the offline stage. And for each case, we propose the corresponding runtime dynamic quality calibration scheme and reconfiguration control policy. Then during runtime, our proposed lightweight quality estimator will evaluate the intermediate quality at specific calibration iteration, which is determined by the novel Markov model-based calibration scheme. If quality violation occurs, the configuration control policy will select the most appropriate approximate computing mode for the following iterations. With the proposed dynamic effort scaling technique, ApproxIt is able to dramatically improve application energy efficiency under quality guarantees, as demonstrated in our experimental results.

A robust graph-based semi-supervised sparse feature selection method

Feature selection is used for excluding redundant features and enhancing learning performance. Abundant unlabeled data are existed in many applications which can be used in semi-supervised feature selection. Semi-supervised sparse feature selection methods have been presented to apply labeled and unlabeled data and consider the correlation among features. Most of the sparse methods utilize square-norm based loss function which is sensitive to outliers. In this paper, we propose a robust Graph-based Semi-Supervised Sparse Feature Selection (GS3FS) method based on the mixed convex and non-convex l2,p-norm (0 < p ≤ 1) minimization on regularization and loss function. The l2,p-norm based regularization ensures that the method selects relevant and sparse features, and the l2,p-norm based loss function makes the method robust to outliers. Proposed method takes the advantages of manifold learning and l2,p-norm minimization on regularization and loss function. The method includes the mixed l2,p-norm which is difficult to solve. To solve the proposed method for the convex (p = 1) and non-convex (0 < p <1) cases, we propose an efficient unified algorithm and prove its convergence. Extensive experiments are carried out on various classification and regression datasets for evaluating the proposed method. The results indicate that the proposed method selects the relevant features and improves the performance of classification and regression models.

ON RANDOM APPROXIMATIONS BY GENERALIZED DISC‐POLYGONS

For two convex discs $K$ and $L$, we say that $K$ is $L$-convex if it is equal to the intersection of all translates of $L$ that contain $K$. In $L$-convexity the set $L$ plays a similar role as closed half-spaces do in the classical notion of convexity. We study the following probability model: Let $K$ and $L$ be $C^2_+$ smooth convex discs such that $K$ is $L$-convex. Select $n$ i.i.d. uniform random points $x_1,\ldots, x_n$ from $K$, and consider the intersection $K_{(n)}$ of all translates of $L$ that contain all of $x_1,\ldots, x_n$. The set $K_{(n)}$ is a random $L$-convex polygon in $K$. We study the expectation of the number of vertices $f_0(K_{(n)})$ and the missed area $A(K\setminus K_{n})$ as $n$ tends to infinity. We consider two special cases of the model. In the first case we assume that the maximum of the curvature of the boundary of $L$ is strictly less than $1$ and the minimum of the curvature of $K$ is larger than $1$. In this setting the expected number of vertices and missed area behave in a similar way as in the classical convex case and in the $r$-spindle convex case (when $L$ is a radius $r$ circular disc). The other case we study is when $K=L$. This setting is special in the sense that an interesting phenomenon occurs: the expected number of vertices tends to a finite limit depending only on $L$. This was previously observed in the special case when $L$ is a circle of radius $r$ (Fodor, Kevei and Vigh (2014)). We also determine the extrema of the limit of the expectation of the number of vertices of $L_{(n)}$ if $L$ is a convex discs of constant width $1$. The formulas we prove can be considered as generalizations of the corresponding $r$-spindle convex statements proved by Fodor, Kevei and Vigh (2014).

On the existence of affine invariant descent directions

This paper begins with a brief review of affine invariance and its significance for iterative algorithms. It then explores the existence of affine invariant descent directions for unconstrained minimization. While there may exist several affine invariant descent directions for smooth functions at a given point, it is shown that for quadratic functions, there exists exactly one invariant descent direction in the strictly convex case and generally none in the case where the Hessian is singular or indefinite. These results can be generalized to smooth nonlinear functions and have implications regarding the initialization of minimization algorithms. They stand in contrast to recent works on constrained convex and nonconvex optimization for which there may exist an affine invariant ‘frame’ that depends on the feasible set and that can be used to define an affine invariant descent direction.

Sparse feature selection in multi-target modeling of carbonic anhydrase isoforms by exploiting shared information among multiple targets

Multi-target modeling can be used for inhibition prediction of CA isoforms, the essential zinc metalloenzymes involved in different biological processes such as tumorigenesis. In this study, the first multi-target model is developed for predicting the activities of inhibitors against CA-I, CA-II, CA-IX, and CA-XII. Structural similarity analysis is carried out for two cancer-related isoforms CA-IX and CA-XII. The mean TM-score value (0.935) reveals a marked similarity between the two structures. To select relevant descriptors for the developed multi-target model, we propose a novel feature selection method based on shared subspace learning, which considers correlation among different targets in multi-target modeling. The proposed shared subspace feature selection method uses the mixed convex and non-convex l2,p-norm (0 ​< ​p ​≤ ​1) minimization on both regularization and loss function to ensure that the loss function is robust to outliers and consider correlation among different descriptors for joint sparse feature selection. To solve the proposed shared subspace feature selection method for convex and non-convex cases, a unified Algorithm is presented. The study utilized a test set to evaluate the performance of the proposed feature selection method with a multi-target kernel smoother model and compare to that of other feature selection methods with the multi-target kernel smoother models. The obtained results demonstrate the superiority of the proposed shared-subspace feature selection approach based on l2,1/2-norm in selecting the most relevant descriptors. Statistical results (RMSEtest ​= ​0.5190, R2test ​= ​0.7613 and Q2ext ​= ​0.7524) demonstrate that the model displays adequate quality for virtual screening. The results also represent the significance of using the shared subspace among different targets in the selection of relevant descriptors to predict the inhibition of CA isoforms.

CONVEXITY CONDITION OF THE GENERALIZED BERNATSKY INTEGRAL FOR ONE SUBCLASS OF STAR-LIKE FUNCTIONS

The article carried out a study on the convexity of the Bernatsky integral in the proposition that the function in question belongs to a subclass of star-shaped functions that satisfy certain conditions. For this, the condition of convexity of univalent functions was considered. The geometrical interpretation of the conditions is given, the radius of the convexity of the star-shaped functions is established. The intervals for the parameter are found for which the Bernatsky integral is a convex function in the whole unit circle, in cases where the parameter does not belong to the given interval, the Bernatsky integral will be a convex function in a circle of smaller radius. Three consequences are given in which various cases of convexity of the Bernatsky integral for analytic functions that belong to classes of functions with certain conditions are analyzed. For the considered classes of analytic functions, the radius of convexity of the Bernatsky integral is determined.

Inexact Variable Metric Stochastic Block-Coordinate Descent for Regularized Optimization

Block-coordinate descent is a popular framework for large-scale regularized optimization problems with block-separable structure. Existing methods have several limitations. They often assume that subproblems can be solved exactly at each iteration, which in practical terms usually restricts the quadratic term in the subproblem to be diagonal, thus losing most of the benefits of higher-order derivative information. Moreover, in contrast to the smooth case, non-uniform sampling of the blocks has not yet been shown to improve the convergence rate bounds for regularized problems. This work proposes an inexact randomized block-coordinate descent method based on a regularized quadratic subproblem, in which the quadratic term can vary from iteration to iteration: a “variable metric.” We provide a detailed convergence analysis for both convex and non-convex problems. Our analysis generalizes, to the regularized case, Nesterov’s proposal for improving convergence of block-coordinate descent by sampling proportional to the blockwise Lipschitz constants. We improve the convergence rate in the convex case by weakening the dependency on the initial objective value. Empirical results also show that significant benefits accrue from the use of a variable metric.

The Dyck bound in the concave 1-dimensional random assignment model

We consider models of assignment for random N blue points and N red points on an interval of length 2N, in which the cost for connecting a blue point in x to a red point in y is the concave function |x − y |p , for 0 < p < 1. Contrarily to the convex case p > 1, where the optimal matching is trivially determined, here the optimization is non-trivial.The purpose of this paper is to introduce a special configuration, that we call the Dyck matching, and to study its statistical properties. We compute exactly the average cost, in the asymptotic limit of large N, together with the first subleading correction. The scaling is remarkable: it is of order N for , order for , and for , and it is universal for a wide class of models. We conjecture that the average cost of the Dyck matching has the same scaling in N as the cost of the optimal matching, and we produce numerical data in support of this conjecture. We hope to produce a proof of this claim in future work.

A Distributed Flexible Delay-Tolerant Proximal Gradient Algorithm

We develop and analyze an asynchronous algorithm for distributed convex optimization when the objective can be written as a sum of smooth functions, local to each worker, and a nonsmooth function. Unlike many existing methods, our distributed algorithm is adjustable to various levels of communication cost, delays, machines' computational power, and functions' smoothness. A unique feature is that the step sizes do not depend on communication delays nor number of machines, which is highly desirable for scalability. We prove that the algorithm converges linearly in the strongly convex case, and provide guarantees of convergence for the non-strongly convex case. The obtained rates are the same as the vanilla proximal gradient algorithm over some introduced epoch sequence that subsumes the delays of the system. We provide numerical results on large-scale machine learning problems to demonstrate the merits of the proposed method.

Primal-Dual Stochastic Gradient Method for Convex Programs with Many Functional Constraints

Stochastic gradient method (SGM) has been popularly applied to solve optimization problems with objective that is stochastic or an average of many functions. Most existing works on SGMs assume that the underlying problem is unconstrained or has an easy-to-project constraint set. In this paper, we consider problems that have a stochastic objective and also many functional constraints. For such problems, it could be extremely expensive to project a point to the feasible set, or even compute subgradient and/or function value of all constraint functions. To find solutions of these problems, we propose a novel (adaptive) SGM based on the classical augmented Lagrangian function. Within every iteration, it inquires a stochastic subgradient of the objective, and a subgradient and the function value of one randomly sampled constraint function. Hence, the per-iteration complexity is low. We establish its convergence rate for convex problems and also problems with strongly convex objective. It can achieve the optimal $O(1/\sqrt{k})$ convergence rate for convex case and nearly optimal $O\big((\log k)/k\big)$ rate for strongly convex case. Numerical experiments on a sample approximation problem of the robust portfolio selection and quadratically constrained quadratic programming are conducted to demonstrate its efficiency.

A Stochastic Line Search Method with Expected Complexity Analysis

For deterministic optimization, line search methods augment algorithms by providing stability and improved efficiency. Here we adapt a classical backtracking Armijo line search to the stochastic optimization setting. While traditional line search relies on exact computations of the gradient and values of the objective function, our method assumes that these values are available up to some dynamically adjusted accuracy which holds with some sufficiently large, but fixed, probability. We bound the expected number of iterations to reach a desired first-order accuracy in the nonconvex, convex, and strongly convex cases and show that this bound matches the complexity bound of deterministic gradient descent up to constants.

Set estimation under biconvexity restrictions

A set in the Euclidean plane is said to be biconvex if, for some angle θ ∈ [0, π∕2), all its sections along straight lines with inclination angles θ and θ + π∕2 are convex sets (i.e., empty sets or segments). Biconvexity is a natural notion with some useful applications in optimization theory. It has also be independently used, under the name of “rectilinear convexity”, in computational geometry. We are concerned here with the problem of asymptotically reconstructing (or estimating) a biconvex set S from a random sample of points drawn on S. By analogy with the classical convex case, one would like to define the “biconvex hull” of the sample points as a natural estimator for S. However, as previously pointed out by several authors, the notion of “hull” for a given set A (understood as the “minimal” set including A and having the required property) has no obvious, useful translation to the biconvex case. This is in sharp contrast with the well-known elementary definition of convex hull. Thus, we have selected the most commonly accepted notion of “biconvex hull” (often called “rectilinear convex hull”): we first provide additional motivations for this definition, proving some useful relations with other convexity-related notions. Then, we prove some results concerning the consistent approximation of a biconvex set S and the corresponding biconvex hull. An analogous result is also provided for the boundaries. A method to approximate, from a sample of points on S, the biconvexity angle θ is also given.

Modeling Planar Joints With Clearance Between the Guide and Roller in Mechanisms

Abstract In mechanisms, the guide and roller form joints with clearance and the roller may make contact with the convex surface or concave surface of the guide. Correctly modeling this type of joint is critical in a mechanism's dynamic performance analysis. This paper proposes a contact kinematic model and a hybrid contact force model for the planar joint and investigates both the convex and concave contact cases. The contact kinematic model is derived from the relative motion between the guide and roller and the hybrid contact force model consists of an elastic force and a damping force. The sliding door mechanism is used as an illustrative example to demonstrate the proposed models. Experimental data are collected for a vehicle's sliding door mechanism. Results show that the proposed models are effective and the simulation results match the general trend of experimental data based on RMS errors.

On a Generalized Langevin Type Nonlocal Fractional Integral Multivalued Problem

We establish sufficient criteria for the existence of solutions for a nonlinear generalized Langevin-type nonlocal fractional-order integral multivalued problem. The convex and non-convex cases for the multivalued map involved in the given problem are considered. Our results rely on Leray–Schauder nonlinear alternative for multivalued maps and Covitz and Nadler’s fixed point theorem. Illustrative examples for the main results are included.