Nonconvex Case Research Articles

Towards sharper excess risk bounds for differentially private pairwise learning

Pairwise learning is a vital part of machine learning. It depends on pairs of training instances, and is naturally fit for modeling relationships between samples. However, as a data driven paradigm, it faces huge privacy issues. Differential privacy (DP) is a useful tool to protect the privacy of machine learning, but corresponding excess population risk bounds are loose in existing DP pairwise learning analysis. In this paper, we propose a gradient perturbation algorithm for pairwise learning to get better risk bounds under Polyak–Łojasiewicz condition, including both convex and non-convex cases. Specifically, for the theoretical risk bound in expectation, previous best results are of rates O(pn2ϵ2+1n) and O(pnϵ+1n) under strongly convex condition and convex conditions, respectively. In this paper, we use the on-average stability and achieve an O(min{pn1.5ϵ+p1.5n2.5ϵ3,pn2ϵ2+1n}) bound, significantly improving previous bounds. For the high probability risk bound, previous best results are analyzed by the uniform stability, and O(βnU+pnϵ) excess population risk bounds are achieved under strongly convex or convex conditions, where βnU is the traditional pairwise uniform stability parameter, it is large since it considers the worst case of the loss sensitivity. In this paper, we propose the pairwise locally elastic stability and improve the high probability bound to O(βEn+pnϵ), in which the pairwise locally elastic stability parameter βE≪βnU because it considers the average sensitivity of the pairwise loss function.

The Proximal Gradient Method for Composite Optimization Problems on Riemannian Manifolds

In this paper, the composite optimization problem is studied on Riemannian manifolds. To tackle this problem, the proximal gradient method to solve composite optimization problems is proposed on Riemannian manifolds. Under some reasonable conditions, the convergence of the proximal gradient method with the backtracking procedure in the nonconvex case is presented. Furthermore, a sublinear convergence rate and the complexity result of the proximal gradient method for convex case are also established on Riemannian manifolds.

Projection free methods on product domains

AbstractProjection-free block-coordinate methods avoid high computational cost per iteration, and at the same time exploit the particular problem structure of product domains. Frank–Wolfe-like approaches rank among the most popular ones of this type. However, as observed in the literature, there was a gap between the classical Frank–Wolfe theory and the block-coordinate case, with no guarantees of linear convergence rates even for strongly convex objectives in the latter. Moreover, most of previous research concentrated on convex objectives. This study now deals also with the non-convex case and reduces above-mentioned theory gap, in combining a new, fully developed convergence theory with novel active set identification results which ensure that inherent sparsity of solutions can be exploited in an efficient way. Preliminary numerical experiments seem to justify our approach and also show promising results for obtaining global solutions in the non-convex case.

Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization

We present a new zero-order optimization method called Minibatch Stochastic Three Points (MiSTP), specifically designed to solve stochastic unconstrained minimization problems when only an approximate evaluation of the objective function is possible. MiSTP is an extension of the Stochastic Three Point Method (STP). The key innovation of MiSTP is that it selects the next point solely based on the objective function approximation, without relying on its exact evaluation. At each iteration, MiSTP generates a random search direction and compares the approximations of the objective function at the current point, the randomly generated direction and its opposite. The best of these three points is chosen as the next iterate. We analyze the worst-case complexity of MiSTP in the convex and non-convex cases and demonstrate that it matches the most accurate complexity bounds known in the literature for zero-order optimization methods. We perform extensive numerical evaluations to assess the computational efficiency of MiSTP and compare its performance to other state-of-the-art methods by testing it on several machine learning tasks. The results show that MiSTP outperforms or has comparable performance against state-of-the-art methods indicating its potential for a wide range of practical applications.

$ V $-Moreau envelope of nonconvex functions on smooth Banach spaces

<p>We continue the study of the properties of the $ V $-Moreau envelope and generalized $ (f, \lambda) $-projection that we started in <sup>[<xref ref-type="bibr" rid="b5">5</xref>]</sup>. In this paper, we study the differentiability and the regularity of the $ V $-Moreau envelope and the Hölder continuity of the generalized $ (f, \lambda) $-projection. Our results extend many existing results from the convex case to the nonconvex case and from Hilbert spaces to Banach spaces. Even on Hilbert spaces and for convex functions and sets, we derived new results.</p>

Powered stochastic optimization with hypergradient descent for large-scale learning systems

Stochastic optimization (SO) algorithms based on the Powerball function, namely powered stochastic optimization (PoweredSO) algorithms, have been confirmed, effectively, and demonstrated great potential in the context of large-scale optimization and machine learning tasks. Nevertheless, the issue of how to determine the learning rate for PoweredSO is a challenge and still unsolved problem. In this paper, we propose a class of adaptive PoweredSO approaches that are efficient, scalable and robust. It takes advantage of the hypergradient descent (HD) technique to automatically acquire an online learning rate for PoweredSO-like methods. In the first part, we study the behavior of the canonical PoweredSO algorithm, the Powerball stochastic gradient descent (pbSGD) method, with HD. The existing PoweredSO algorithms also suffer from the high variance because they take the similar algorithmic framework to SO algorithms, arising from sampling tactics. Therefore, the second portion develops an adaptive powered variance-reduced optimization method via utilizing both variance-reduced technique and HD. Moreover, we present the convergence analysis of the proposed algorithms and explore their iteration complexity on non-convex cases. Numerical experiments are conducted on machine learning tasks, verifying the superior performance over modern SO algorithms.

Determining the area of a technical system stability in fuzzy sets terms

Ensuring the stability of the technical systems functioning, including those in water transport, is an important area for the application of mathematical methods, including probabilistic and mathematical-statistical methods. The stochastic nature of disturbances affecting complex systems and worsening the level of stability of the latter should be reflected in the implemented human-machine control loops. One of the directions here is the accounting and use of (usually competing) expert assessments, on the one hand, and a reasonable definition of stability indicators, a choice from a variety of possibilities, on the other hand. From a general point of view of management theory, the method of using expert assessments developed in the paper can be considered as an integral part of the risk management problem that is being developed so far, mainly for economic systems and using only financial criteria. The concept of the minimum allowable efficiency of a technical system is introduced in the paper. On its basis, an approach in which the indicator of the stability of a technical system is linked to a system-wide criterion indicator of the quality of its functioning is implemented. The corresponding mathematical model as a problem of mathemati- cal programming is formulated. Approaches to its solution including for the nonconvex multiextremal case are discussed. A feature of the proposed approach to the definition of the integral indicator of the stability of the dynamics of a technical system is the use of expert assessments in determining the probable range of changes in the controlled variables of the optimization model, implemented as a mathematical programming problem, which logically leads to the use of fuzzy set theory. With the non-linearity of the objective functions and restrictions that arise in this case, the gradient method of searching for a conditional optimum is effective.

Stability-Based Generalization Analysis of the Asynchronous Decentralized SGD

The generalization ability often determines the success of machine learning algorithms in practice. Therefore, it is of great theoretical and practical importance to understand and bound the generalization error of machine learning algorithms. In this paper, we provide the first generalization results of the popular stochastic gradient descent (SGD) algorithm in the distributed asynchronous decentralized setting. Our analysis is based on the uniform stability tool, where stable means that the learned model does not change much in small variations of the training set. Under some mild assumptions, we perform a comprehensive generalizability analysis of the asynchronous decentralized SGD, including generalization error and excess generalization error bounds for the strongly convex, convex, and non-convex cases. Our theoretical results reveal the effects of the learning rate, training data size, training iterations, decentralized communication topology, and asynchronous delay on the generalization performance of the asynchronous decentralized SGD. We also study the optimization error regarding the objective function values and investigate how the initial point affects the excess generalization error. Finally, we conduct extensive experiments on MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets to validate the theoretical findings.

Federated learning with superquantile aggregation for heterogeneous data

We present a federated learning framework that is designed to robustly deliver good predictive performance across individual clients with heterogeneous data. The proposed approach hinges upon a superquantile-based learning objective that captures the tail statistics of the error distribution over heterogeneous clients. We present a stochastic training algorithm that interleaves differentially private client filtering with federated averaging steps. We prove finite time convergence guarantees for the algorithm: $$O(1/\sqrt{T})$$ in the nonconvex case in T communication rounds and $$O(\exp (-T/\kappa ^{3/2}) + \kappa /T)$$ in the strongly convex case with local condition number $$\kappa$$ . Experimental results on benchmark datasets for federated learning demonstrate that our approach is competitive with classical ones in terms of average error and outperforms them in terms of tail statistics of the error.

Exploring the systematic coupling coordination degrees of innovation activities in China’s high-tech industry

ABSTRACT The high-tech industry plays an important role in promoting the upgrading of industrial structures. Its innovation activities show an obvious two-system structure. To promote co-development and improve industrial competitiveness, it is essential to examine the efficiency and measure the coupling coordination degree between two systems. With this regard, we first measure the relative efficiency using nonparametric technologies under different specifications of each system. Second, the coupling degree and coupling coordination degree between two systems have been calculated. Finally, comparative analyses of efficiency and coupling coordination degrees have been analysed from the perspective of the regional high-tech industry. Empirical results indicate the low overall technical efficiency in the regional high-tech industry is caused by low-scale efficiency under nonconvex. However, it is caused by low pure technical efficiency under convex. Moreover, the average efficiency of technology development in regional high-tech industry is higher than that of technology transformation for convex and nonconvex cases. Furthermore, the degree of coupling coordination of the two systems of the high-tech industry in all regions is moderate and above coordination. The empirical results can provide useful suggestions for policymakers to create an environment conducive to industrial development.

Stability analysis of stochastic gradient descent for homogeneous neural networks and linear classifiers

We prove new generalization bounds for stochastic gradient descent when training classifiers with invariances. Our analysis is based on the stability framework and covers both the convex case of linear classifiers and the non-convex case of homogeneous neural networks. We analyze stability with respect to the normalized version of the loss function used for training. This leads to investigating a form of angle-wise stability instead of euclidean stability in weights. For neural networks, the measure of distance we consider is invariant to rescaling the weights of each layer. Furthermore, we exploit the notion of on-average stability in order to obtain a data-dependent quantity in the bound. This data-dependent quantity is seen to be more favorable when training with larger learning rates in our numerical experiments. This might help to shed some light on why larger learning rates can lead to better generalization in some practical scenarios.

Hessian-based semi-supervised feature selection using generalized uncorrelated constraint

Feature selection (FS) aims to eliminate redundant features and choose the informative ones. Since labeled data are not always easily available and abundant unlabeled data are often accessible, the importance of semi-supervised FS (SSFS), which selects the informative features utilizing the labeled and unlabeled data, becomes apparent. Semi-supervised sparse FS based on graph Laplacian (GL) regularization has become a popular technique. The GL regularization biases the solution towards a constant geodesic function, has a poor extrapolating ability, and cannot preserve well the topological structure of data. Traditional ridge regression is widely utilized by the GL-based semi-supervised sparse FS, but the results cannot gain the closed-form solution and carry out the manifold structure. To tackle these problems, we propose a Hessian-based SSFS framework using the generalized uncorrelated constraint (HSFSGU) and the mixed ℓ2,p-norm (0<p≤1) regularization. We also propose a Hessian–Laplacian-based SSFS framework using the generalized uncorrelated constraint (HLSFSGU) which utilizes the combination of GL and Hessian matrices. Our frameworks use the Hessian regularization for maintaining the topological structure of data well, the ℓ2,p-norm regularization for making the projection matrix appropriate for FS, and the generalized uncorrelated constraint for preventing exceeding row sparsity of the projection matrix and providing the elegant closed-form solution for our frameworks. We also present a unified algorithm to solve the frameworks for convex and non-convex cases, and prove its convergence. The results of the experiments depict the ability of the Hessian-based frameworks under generalized uncorrelated constraint in selecting the informative features.

A limited memory Quasi-Newton approach for multi-objective optimization

In this paper, we deal with the class of unconstrained multi-objective optimization problems. In this setting we introduce, for the first time in the literature, a Limited Memory Quasi-Newton type method, which is well suited especially in large scale scenarios. The proposed algorithm approximates, through a suitable positive definite matrix, the convex combination of the Hessian matrices of the objectives; the update formula for the approximation matrix can be seen as an extension of the one used in the popular L-BFGS method for scalar optimization. Equipped with a Wolfe type line search, the considered method is proved to be well defined even in the nonconvex case. Furthermore, for twice continuously differentiable strongly convex problems, we state global and R-linear convergence to Pareto optimality of the sequence of generated points. The performance of the new algorithm is empirically assessed by a thorough computational comparison with state-of-the-art Newton and Quasi-Newton approaches from the multi-objective optimization literature. The results of the experiments highlight that the proposed approach is generally efficient and effective, outperforming the competitors in most settings. Moreover, the use of the limited memory method results to be beneficial within a global optimization framework for Pareto front approximation.

Randomized block-coordinate adaptive algorithms for nonconvex optimization problems

Nonconvex optimization problems have always been one focus in deep learning, in which many fast adaptive algorithms based on momentum are applied. However, the full gradient computation of high-dimensional feature vector in the above tasks become prohibitive. To reduce the computation cost for optimizers on nonconvex optimization problems typically seen in deep learning, this work proposes a randomized block-coordinate adaptive optimization algorithm, named RAda, which randomly picks a block from the full coordinates of the parameter vector and then sparsely computes its gradient. We prove that RAda converges to a δ-accurate solution with the stochastic first-order complexity of O(1/δ2), where δ is the upper bound of the gradient’s square, under nonconvex cases. Experiments on public datasets including CIFAR-10, CIFAR-100, and Penn TreeBank, verify that RAda outperforms the other compared algorithms in terms of the computational cost.

A local spline regression-based framework for semi-supervised sparse feature selection

Feature selection (FS) is extensively applied in many machine learning applications for the selection of relevant features from data sets. A lot of unlabeled data are available in a variety of applications that can be exploited for semi-supervised FS to address the lack of labeled data and improve learning performance. Recently, semi-supervised sparse FS based on graph Laplacian has obtained considerable research interest, which uses the correlation between features in the process of FS. However, the Laplacian regularization has a weak extrapolating power and a bias towards the constant geodesic function, and cannot retain the local topology well. In this paper, a spline regression-based framework for semi-supervised sparse FS (SRS3FS) is proposed, which uses the mixed convex and non-convex ℓ2,p-norm (0 <p≤1) regularization to select the relevant features and consider the correlation between features. The framework exploits local spline regression to retain the geometry structure of labeled and unlabeled data and encodes the data distribution. A unified iterative algorithm is presented to solve the proposed framework for the convex and non-convex cases, and its convergence is theoretically and experimentally proved. Experiments on several data sets illustrate the effectiveness of our framework in the selection of the most relevant and discriminative features.

Feature screening strategy for non-convex sparse logistic regression with log sum penalty

L1 logistic regression is an efficient classification algorithm. Leveraging on the convexity of the model and the sparsity of L1 norm, techniques named safe screening rules can help accelerate solvers for high-dimensional data. Recently, Log Sum Penalty (LSP) provides better theoretical guarantees in identifying relevant variables, and sparse logistic regression with LSP has been proposed. However, due to the non-convexity of this model, the training process is time-consuming. Furthermore, the existing safe screening rules cannot be directly applied to accelerate it. To deal with this issue, in this paper, based on the iterative majorization minimization (MM) principle, we construct a novel method that can effectively save training time. To do this, we first design a feature screening strategy for the inner solver and then build another rule to propagate screened features between the iterations of MM. After that, we introduce a modified feature screening strategy to further accelerate the computational speed, which can obtain a smaller safe region thanks to reconstructing the strong global concavity bound. Moreover, our rules can be applied to other non-convex cases. Experiments on nine benchmark datasets verify the effectiveness and security of our algorithm.

Bootstrapping globally optimal variational calculus solutions

Whereas in a coordinate-dependent setting the Euler–Lagrange equations establish necessary conditions for solving variational problems in which both the integrands of functionals and the resulting paths are assumed to be sufficiently smooth, uniqueness and global optimality are generally hard to prove in the absence of convexity conditions, and often times they may not even exist. This is also true for variational problems on Lie groups, with the Euler–Poincaré equation establishing necessary conditions. The difficulties compound when integrands and/or the optimal paths are not sufficiently regular, since in this case the classical necessary conditions no longer apply. This article therefore reviews several nonstandard cases where unique globally optimal solutions can be guaranteed, and establishes a “bootstrapping” process to build new globally optimal variational solutions on larger spaces from existing ones on smaller spaces. Surprisingly, it is possible to prove global optimality in some nonconvex cases where even the regularity conditions required for classical necessary conditions do not hold. This general theory is then applied to several topics such as optimal framing of curves in three-dimensional Euclidean space, optimal motion interpolation, and optimal reparametrization of video sequences to compare salient actions.

Operator theoretic approach to optimal control problems characterized by the Caputo fractional differential equations

In this paper, our aim is to solve an optimal control problem by using an operator theoretic approach in the sense of Caputo fractional derivative. We first reduce the fractional dynamical system into an equivalent Hammerstein operator equation and then we provide a sufficient condition for the operators to ensure the existence of an optimal pair for the fractional dynamical system. By analyzing minimum conditions for the fractional optimal control problem (FOCP), the existence results of an optimal pair are obtained for both convex and non-convex cases. We derive an optimality system with a quadratic cost functional by using the Gateaux derivative. Finally, we relate our optimality system with the Hamiltonian system of minimum principle, and some examples are included to demonstrate the effectiveness of theoretical results. The primary motivation for this research is to investigate a new approach to solving the fractional optimal control problem employing functional analysis and operator theory techniques.

Momentum-Based Variance-Reduced Proximal Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization

Stochastic gradient methods (SGMs) have been extensively used for solving stochastic problems or large-scale machine learning problems. Recent works employ various techniques to improve the convergence rate of SGMs for both convex and nonconvex cases. Most of them require a large number of samples in some or all iterations of the improved SGMs. In this paper, we propose a new SGM, named PStorm, for solving nonconvex nonsmooth stochastic problems. With a momentum-based variance reduction technique, PStorm can achieve the optimal complexity result $$O(\varepsilon ^{-3})$$ to produce a stochastic $$\varepsilon $$ -stationary solution, if a mean-squared smoothness condition holds. Different from existing optimal methods, PStorm can achieve the $${O}(\varepsilon ^{-3})$$ result by using only one or O(1) samples in every update. With this property, PStorm can be applied to online learning problems that favor real-time decisions based on one or O(1) new observations. In addition, for large-scale machine learning problems, PStorm can generalize better by small-batch training than other optimal methods that require large-batch training and the vanilla SGM, as we demonstrate on training a sparse fully-connected neural network and a sparse convolutional neural network.

A memetic procedure for global multi-objective optimization

In this paper we consider multi-objective optimization problems over a box. Several computational approaches to solve these problems have been proposed in the literature, that broadly fall into two main classes: evolutionary methods, which are usually very good at exploring the feasible region and retrieving good solutions even in the nonconvex case, and descent methods, which excel in efficiently approximating good quality solutions. In this paper, first we confirm, through numerical experiments, the advantages and disadvantages of these approaches. Then we propose a new method which combines the good features of both. The resulting algorithm, which we call Non-dominated Sorting Memetic Algorithm, besides enjoying interesting theoretical properties, excels in all of the numerical tests we performed on several, widely employed, test functions.