Difference Of Convex Functions Research Articles

Online Stochastic DCA With Applications to Principal Component Analysis.

Stochastic algorithms are well-known for their performance in the era of big data. In this article, we study nonsmooth stochastic Difference-of-Convex functions (DC) programs-the major class of nonconvex stochastic optimization, which have a variety of applications in divers domains, in particular, machine learning. We propose new online stochastic algorithms based on the state-of-the-art DC Algorithm (DCA)-a powerful approach in nonconvex programming framework, in the online context of streaming data continuously generated by some (unknown) source distribution. The new schemes use the stochastic approximations (SAs) principle: deterministic quantities of the standard DCA are replaced by their noisy estimators constructed using newly arriving samples. The convergence analysis of the proposed algorithms is studied intensively with the help of tools from modern convex analysis and martingale theory. Finally, we study several aspects of the proposed algorithms on an important problem in machine learning: the expected problem in principal component analysis (PCA).

An inexact proximal linearized DC algorithm with provably terminating inner loop

Standard approaches to difference-of-convex (DC) programs require exact solution to a convex subproblem at each iteration, which generally requires noiseless computation and infinite iterations of an inner iterative algorithm. To tackle these difficulties, inexact DC algorithms have been proposed, mostly by relaxing the convex subproblem to an approximate monotone inclusion problem. However, there is no guarantee that such relaxation can lead to a finitely terminating inner loop. In this paper, we point out the termination issue of existing inexact DC algorithms by presenting concrete counterexamples. Exploiting the notion of ϵ-subdifferential, we propose a novel inexact proximal linearized DC algorithm termed tPLDCA. Despite permission to a great extent of inexactness in computation, tPLDCA enjoys the same convergence guarantees as exact DC algorithms. Most noticeably, the inner loop of tPLDCA is guaranteed to terminate in finite iterations as long as the inner iterative algorithm converges to a solution of the proximal point subproblem, which makes an essential difference from prior arts. In addition, by assuming the first convex component of the DC function to be pointwise maximum of finitely many convex smooth functions, we propose a computational friendly surrogate of ϵ-subdifferential, whereby we develop a feasible implementation of tPLDCA. Numerical results demonstrate the effectiveness of the proposed implementation of tPLDCA.

Difference of Convex Functions Programming With Machine-Learning Prior for the Imaging Problem in Electrical Capacitance Tomography.

The electrical capacitance tomography technology has potential benefits for the process industry by providing visualization of material distributions. One of the main technical gaps and impediments that must be overcome is the low-quality tomogram. To address this problem, this study introduces the data-guided prior and combines it with the electrical measurement mechanism and the sparsity prior to produce a new difference of convex functions programming problem that turns the image reconstruction problem into an optimization problem. The data-guided prior is learned from a provided dataset and captures the details of imaging targets since it is a specific image. A new numerical scheme that allows a complex optimization problem to be split into a few less difficult subproblems is developed to solve the challenging difference of convex functions programming problem. A new dimensionality reduction method is developed and combined with the relevance vector machine to generate a new learning engine for the forecast of the data-guided prior. The new imaging method fuses multisource information and unifies data-guided and measurement physics modeling paradigms. Performance evaluation results have validated that the new method successfully works on a series of test tasks with higher reconstruction quality and lower noise sensitivity than the popular imaging methods.

Markov chain stochastic DCA and applications in deep learning with PDEs regularization

This paper addresses a large class of nonsmooth nonconvex stochastic DC (difference-of-convex functions) programs where endogenous uncertainty is involved and i.i.d. (independent and identically distributed) samples are not available. Instead, we assume that it is only possible to access Markov chains whose sequences of distributions converge to the target distributions. This setting is legitimate as Markovian noise arises in many contexts including Bayesian inference, reinforcement learning, and stochastic optimization in high-dimensional or combinatorial spaces. We then design a stochastic algorithm named Markov chain stochastic DCA (MCSDCA) based on DCA (DC algorithm) - a well-known method for nonconvex optimization. We establish the convergence analysis in both asymptotic and nonasymptotic senses. The MCSDCA is then applied to deep learning via PDEs (partial differential equations) regularization, where two realizations of MCSDCA are constructed, namely MCSDCA-odLD and MCSDCA-udLD, based on overdamped and underdamped Langevin dynamics, respectively. Numerical experiments on time series prediction and image classification problems with a variety of neural network topologies show the merits of the proposed methods.

Fused robust geometric nonparallel hyperplane support vector machine for pattern classification

Recently, introducing nonconvex loss functions in support vector machine (SVM) to improve the robustness against varies noises has been drawing much attention. In this paper, we first construct a new robust capped asymmetric elastic net (CaEN) loss function. Second, we describe a novel robust Huberized kernel-based (HK) loss function and theoretically demonstrate several important properties, such as smoothness, boundness and the trade-off between the standard least squares and the truncated least squares. Finally, we apply the CaEN loss and the HK loss into elastic net nonparallel hyperplane SVM (ENNHSVM) to develop a fused robust geometric nonparallel SVM (FRGNHSVM). The proposed FRGNHSVM not only inherits the advantages of ENNHSVM but also improves the robustness of classification problems. An efficient Pegasos-based DC (difference of convex functions) algorithm is implemented to solve the FRGNHSVM optimization problem. In comparison with four famous SVMs, including Lagrangian SVM, twin SVM, pinball SVM and C-loss twin SVM, experimental results on simulations and twelve UCI datasets show that the proposed FRGNHSVM can often improve more than 5% average prediction accuracy. Moreover, the performance of the high prediction accuracy of FRGHNSVM is more significant along with the ratio of label noise increasing, indicating its superiority in dealing with label-contaminated datasets.

Cardinality optimization in constraint-based modelling: application to human metabolism.

Several applications in constraint-based modelling can be mathematically formulated as cardinality optimization problems involving the minimization or maximization of the number of nonzeros in a vector. These problems include testing for stoichiometric consistency, testing for flux consistency, testing for thermodynamic flux consistency, computing sparse solutions to flux balance analysis problems and computing the minimum number of constraints to relax to render an infeasible flux balance analysis problem feasible. Such cardinality optimization problems are computationally complex, with no known polynomial time algorithms capable of returning an exact and globally optimal solution. By approximating the zero-norm with nonconvex continuous functions, we reformulate a set of cardinality optimization problems in constraint-based modelling into a difference of convex functions. We implemented and numerically tested novel algorithms that approximately solve the reformulated problems using a sequence of convex programs. We applied these algorithms to various biochemical networks and demonstrate that our algorithms match or outperform existing related approaches. In particular, we illustrate the efficiency and practical utility of our algorithms for cardinality optimization problems that arise when extracting a model ready for thermodynamic flux balance analysis given a human metabolic reconstruction. Open source scripts to reproduce the results are here https://github.com/opencobra/COBRA.papers/2023_cardOpt with general purpose functions integrated within the COnstraint-Based Reconstruction and Analysis toolbox: https://github.com/opencobra/cobratoolbox.

Minimizing Compositions of Differences-of-Convex Functions with Smooth Mappings

We address the so-called DC (difference-of-convex functions) composite minimization problems (or DC composite programs) whose objective function is a composition of a DC function with a continuously differentiable mapping. We first develop an algorithm named DC composite algorithm (DCCA in short) for unconstrained DC composite programs and further extend to DC composite programs with constraints of inclusion associated with a smooth mapping and a closed convex set. The convergence analysis of the proposed algorithms is investigated. Applications of DCCA for two different problems, computation of the numerical radius of a square matrix and minimization of composite energies, are presented.

Optimality Conditions in DC-Constrained Mathematical Programming Problems

This paper provides necessary and sufficient optimality conditions for abstract-constrained mathematical programming problems in locally convex spaces under new qualification conditions. Our approach exploits the geometrical properties of certain mappings, in particular their structure as difference of convex functions, and uses techniques of generalized differentiation (subdifferential and coderivative). It turns out that these tools can be used fruitfully out of the scope of Asplund spaces. Applications to infinite, stochastic and semi-definite programming are developed in separate sections.

On Fenchel c-conjugate dual problems for DC optimization: characterizing weak, strong and stable strong duality

In this paper we present two Fenchel-type dual problems for a DC (difference of convex functions) optimization primal one. They have been built by means of the c-conjugation scheme, a pattern of conjugation which has been shown to be suitable for evenly convex functions. We study characterizations of weak, strong and stable strong duality for both pairs of primal–dual problems. We also give conditions which relate the existence of strong and stable strong duality for both pairs.

A generalized adaptive robust distance metric driven smooth regularization learning framework for pattern recognition

In this work, we propose a novel generalized adaptive robust distance metric called Lδ(u). Compared with other distance metrics, Lδ(u) has some desirable salient properties, such as symmetry, boundedness, robustness, nonconvexity, and adaptivity, with both first-order and higher-order moments from samples. On the other hand, Lδ(u) can pick different robust distance metrics for different learning tasks during the learning process by the adaptive parameter δ. Furthermore, we apply Lδ(u) to twin extreme learning machine (TELM) and develop an smooth regularized TELM learning framework for supervised and semi-supervised classification. By introducing the structural risk minimization (SRM) principle and smoothing techniques, the learning framework perfectly overcomes the computational burden associated with the matrix inversion operation required during TELM solving, while also significantly improving performance. More importantly, the proposed learning framework not only improves the robustness of TELM, but also can effectively use the geometric information of the marginal distribution embedded in the unlabeled samples to construct a more reasonable classifier. Finally, the globally convergent and quadratic convergent fast Newton-Armijo algorithm and DC (difference of convex functions) programming algorithm (DCA) are designed to solve the proposed methods. Experimental results demonstrate the effectiveness and robustness of our methods.

Sparse Network Identification for Consensus Discrete-Time Systems

This paper investigates an identification problem of sparse topology among nodes for consensus networked control systems. We suppose the system is modeled by a discrete-time first-order consensus network stochastically disturbed by white gaussian process noise. Our proposed method employs a sparsity-penalized maximum likelihood approach, where edge weights are penalized by the ℓ1 norm to address the sparsity of network topology. The optimization problem is shown to be expressed as a DC (Difference of Convex functions) optimization problem. A numerical example illustrates that the proposed method is useful to locate edges and to estimate their weights.

Impulse-to-Peak Optimization of Positive Linear Systems via DC Programming

This paper is concerned with the optimization of the impulse-to-peak value of positive linear systems. Although the results of the analysis of the peak value of positive linear systems have been reported in the literature, the synthesis problem of impulse-to-peak value remains unresolved. To address this problem, in this paper we show that a cost-constrained minimization problem of the impulse-to-peak value of parameterized positive linear systems can be reduced to a DC (difference of convex functions) program. The derivation utilizes the log-log convexity of posynomial functions and a recent characterization of the impulse-to-peak value of positive systems via a linear program. To illustrate the effectiveness of our results, we study the optimization problem for minimizing the peak infection of networked epidemic spreading processes.

Algorithms for Difference-of-Convex Programs Based on Difference-of-Moreau-Envelopes Smoothing

In this paper, we consider minimization of a difference-of-convex (DC) function with and without linear equality constraints. We first study a smooth approximation of a generic DC function, termed difference-of-Moreau-envelopes (DME) smoothing, where both components of the DC function are replaced by their respective Moreau envelopes. The resulting smooth approximation is shown to be Lipschitz differentiable, capture stationary points, local, and global minima of the original DC function, and enjoy some growth conditions, such as level-boundedness and coercivity, for broad classes of DC functions. For a smoothed DC program without linear constraints, it is shown that the classic gradient descent method and an inexact variant converge to a stationary solution of the original DC function in the limit with a rate of [Formula: see text], where K is the number of proximal evaluations of both components. Furthermore, when the DC program is explicitly constrained in an affine subspace, we combine the smoothing technique with the augmented Lagrangian function and derive two variants of the augmented Lagrangian method (ALM), named linearly constrained DC (LCDC)-ALM and composite LCDC-ALM, targeting on different structures of the DC objective function. We show that both algorithms find an ϵ-approximate stationary solution of the original DC program in [Formula: see text] iterations. Comparing to existing methods designed for linearly constrained weakly convex minimization, the proposed ALM-based algorithms can be applied to a broader class of problems, where the objective contains a nonsmooth concave component. Finally, numerical experiments are presented to demonstrate the performance of the proposed algorithms. Funding: This work was partially supported by the NSF [Grant ECCS1751747]. Supplemental Material: The e-companion is available at https://doi.org/10.1287/ijoo.2022.0087 .

Secure Energy Efficiency Maximization for Untrusted Wireless-Powered Full-Duplex Relay Networks Under Nonlinear Energy Harvesting

In this article, we raise the concern on secure energy efficiency (SEE) over the wireless-powered untrusted relay networks with nonlinear energy harvesting (EH). To achieve data confidentiality and secrecy performance enhancement concurrently, the untrusted relay can enable simultaneous energy signal reception and confidential information forwarding during the second phase, where no dedicated energy transmission phase or relaying signal splitting is needed for the energy supply. In particular, with the help of destination jamming, the relay can also harvest energy from source signal and jamming signal during the first phase, which is different from conventional relaying protocol that only harvests energy from source and recycled energy. Considering the nonlinear EH efficiency, our objective is to maximize the SEE through piecewise region selection and power allocation. To address this intractable nonconvex problem, a two-layer algorithm based on optimization decoupling, fractional programming, Lagrange dual decomposition and difference of convex functions (DC) programming is proposed. The core idea of the proposed algorithm is to convert the primary problem into simple subproblems by sequentially exploiting the above-mentioned optimization methods. Simulation results reveal the superior secrecy performance of the proposed scheme. The proposed algorithm achieves near-optimal solutions with faster convergence rate.

Energy efficient MIMO–NOMA aided IoT network in B5G communications

To accelerate future intelligent wireless systems , we designed an energy-efficient Massive multiple-input-multiple-output (MIMO)- non-orthogonal multiple access (NOMA) aided internet of things (IoT) network in this paper to support the massive number of distributed users and IoT devices with seamless data transfer and maintain connectivity between them. Massive MIMO has been identified as a suitable technology to implement the energy efficient IoT network in beyond 5G (B5G) communications due to its distinct characteristics with large number of antennas. However, to provide fast data transfer and maintain hyper connectivity between the IoT devices in B5G communications will bring the challenge of energy deficiency. Hence, we considered a massive MIMO–NOMA aided IoT network considering imperfect channel state information and practical power consumption at the transmitter. The far users of the base stations are selected to investigate the power consumption and quality of service. Then, calculate the power consumption which is non-convex function and non-deterministic polynomial problem. To solve the above problem, fractional programming properties are applied which converted polynomial problem into the difference of convex function . And then we employed the successive convex approximation technique to represent the non-convex to convex function. Effective iterative based branch and the reduced bound process are utilized to solve the problem. Numerical results observe that our implemented approach surpasses previous standard algorithms on the basis of convergence, energy-efficiency and user fairness.

Difference of convex algorithms for bilevel programs with applications in hyperparameter selection

In this paper, we present difference of convex algorithms for solving bilevel programs in which the upper level objective functions are difference of convex functions, and the lower level programs are fully convex. This nontrivial class of bilevel programs provides a powerful modelling framework for dealing with applications arising from hyperparameter selection in machine learning. Thanks to the full convexity of the lower level program, the value function of the lower level program turns out to be convex and hence the bilevel program can be reformulated as a difference of convex bilevel program. We propose two algorithms for solving the reformulated difference of convex program and show their convergence to stationary points under very mild assumptions. Finally we conduct numerical experiments to a bilevel model of support vector machine classification.

Stochastic Difference-of-Convex-Functions Algorithms for Nonconvex Programming

The paper deals with stochastic difference-of-convex-functions (DC) programs, that is, optimization problems whose cost function is a sum of a lower semicontinuous DC function and the expectation of a stochastic DC function with respect to a probability distribution. This class of nonsmooth and nonconvex stochastic optimization problems plays a central role in many practical applications. Although there are many contributions in the context of convex and/or smooth stochastic optimization, algorithms dealing with nonconvex and nonsmooth programs remain rare. In deterministic optimization literature, the DC algorithm (DCA) is recognized to be one of the few algorithms able to effectively solve nonconvex and nonsmooth optimization problems. The main purpose of this paper is to present some new stochastic DCAs for solving stochastic DC programs. The convergence analysis of the proposed algorithms is carefully studied, and numerical experiments are conducted to justify the algorithms' behaviors.

A New Boosted Proximal Point Algorithm for Minimizing Nonsmooth DC Functions

Several optimization schemes have been known for convex optimization problems. A significant progress to go beyond convexity was made by considering the class of functions representable as difference of convex functions which constitute the backbone of nonconvex programming and global optimization. In this article, we introduce new algorithm to minimize the difference of a continuously differentiable function and a convex function that accelerate the convergence of the classical proximal point algorithm. We prove that the point computed by proximal point algorithm can be used to define a descent direction for the objective function evaluated at this point. Our algorithms are based on a combination of proximal point algorithm together with a line search step that uses this descent direction. Convergence of the algorithms is proved and the rate of convergence is analyzed under the strong Kurdyka–Łojasiewicz property of the objective function.

DCA for Sparse Quadratic Kernel-Free Least Squares Semi-Supervised Support Vector Machine

With the development of science and technology, more and more data have been produced. For many of these datasets, only some of the data have labels. In order to make full use of the information in these data, it is necessary to classify them. In this paper, we propose a strong sparse quadratic kernel-free least squares semi-supervised support vector machine (SSQLSS3VM), in which we add a ℓ0norm regularization term to make it sparse. An NP-hard problem arises since the proposed model contains the ℓ0 norm and another nonconvex term. One important method for solving the nonconvex problem is the DC (difference of convex function) programming. Therefore, we first approximate the ℓ0 norm by a polyhedral DC function. Moreover, due to the existence of the nonsmooth terms, we use the sGS-ADMM to solve the subproblem. Finally, empirical numerical experiments show the efficiency of the proposed algorithm.

A Unifying Framework of High-Dimensional Sparse Estimation with Difference-of-Convex (DC) Regularizations

Under the linear regression framework, we study the variable selection problem when the underlying model is assumed to have a small number of nonzero coefficients. Nonconvex penalties in specific forms are well studied in the literature for sparse estimation. Recent work pointed out that nearly all existing nonconvex penalties can be represented as difference-of-convex (DC) functions, which are the difference of two convex functions, while itself may not be convex. There is a large existing literature on optimization problems when their objectives and/or constraints involve DC functions. Efficient numerical solutions have been proposed. Under the DC framework, directional-stationary (d-stationary) solutions are considered, and they are usually not unique. In this paper, we show that under some mild conditions, a certain subset of d-stationary solutions in an optimization problem (with a DC objective) has some ideal statistical properties: namely, asymptotic estimation consistency, asymptotic model selection consistency, asymptotic efficiency. Our assumptions are either weaker than or comparable with those conditions that have been adopted in other existing works. This work shows that DC is a nice framework to offer a unified approach to these existing works where nonconvex penalties are involved. Our work bridges the communities of optimization and statistics.