Difference Of Convex Functions Programming Research Articles

DC programming and DCA for solving Brugnano–Casulli piecewise linear systems

Piecewise linear optimization is one of the most frequently used optimization models in practice, such as transportation, finance and supply-chain management. In this paper, we investigate a particular piecewise linear optimization that is optimizing the norm of piecewise linear functions (NPLF). Specifically, we are interested in solving a class of Brugnano–Casulli piecewise linear systems (PLS), which can be reformulated as an NPLF problem. Speaking generally, the NPLF is considered as an optimization problem with a nonsmooth, nonconvex objective function. A new and efficient optimization approach based on DC (Difference of Convex functions) programming and DCA (DC Algorithms) is developed. With a suitable DC formulation, we design a DCA scheme, named ℓ1-DCA, for the problem of optimizing the ℓ1-norm of NPLF. Thanks to particular properties of the problem, we prove that under some conditions, our proposed algorithm converges to an exact solution after a finite number of iterations. In addition, when a nonglobal solution is found, a numerical procedure is introduced to find a feasible point having a smaller objective value and to restart ℓ1-DCA at this point. Several numerical experiments illustrate these interesting convergence properties. Moreover, we also present an application to the free-surface hydrodynamic problem, where the correct numerical modeling often requires to have the solution of special PLS, with the aim of showing the efficiency of the proposed method.

A sparse extreme learning machine framework by continuous optimization algorithms and its application in pattern recognition

Extreme learning machine (ELM) has demonstrated great potential in machine learning owing to its simplicity, rapidity and good generalization performance. In this investigation, based on least-squares estimate (LSE) and least absolute deviation (LAD), we propose four sparse ELM formulations with zero-norm regularization to automatically choose the optimal hidden nodes. Furthermore, we develop two continuous optimization methods to solve the proposed problems respectively. The first is DC (difference of convex functions) approximation approach that approximates the zero-norm by a DC function, and the resulting optimizations are posed as DC programs. The second is an exact penalty technique for zero-norm, and the resulting problems are reformulated as DC programs, and the corresponding DCAs converge finitely. Moreover, the proposed framework is applied directly to recognize the hardness of licorice seeds using near-infrared spectral data. Experiments in different spectral regions illustrate that the proposed approaches can reduce the number of hidden nodes (or output features), while either improve or show no significant difference in generalization compared with the traditional ELM methods and support vector machine (SVM). Experiments on several benchmark data sets demonstrate that the proposed framework is competitive with the traditional approaches in generalization, but selects fewer output features.

Efficient Nonnegative Matrix Factorization by DC Programming and DCA.

In this letter, we consider the nonnegative matrix factorization (NMF) problem and several NMF variants. Two approaches based on DC (difference of convex functions) programming and DCA (DC algorithm) are developed. The first approach follows the alternating framework that requires solving, at each iteration, two nonnegativity-constrained least squares subproblems for which DCA-based schemes are investigated. The convergence property of the proposed algorithm is carefully studied. We show that with suitable DC decompositions, our algorithm generates most of the standard methods for the NMF problem. The second approach directly applies DCA on the whole NMF problem. Two algorithms-one computing all variables and one deploying a variable selection strategy-are proposed. The proposed methods are then adapted to solve various NMF variants, including the nonnegative factorization, the smooth regularization NMF, the sparse regularization NMF, the multilayer NMF, the convex/convex-hull NMF, and the symmetric NMF. We also show that our algorithms include several existing methods for these NMF variants as special versions. The efficiency of the proposed approaches is empirically demonstrated on both real-world and synthetic data sets. It turns out that our algorithms compete favorably with five state-of-the-art alternating nonnegative least squares algorithms.

A new semi-supervised classifier based on maximum vector-angular margin

Semi-supervised learning is an attractive method in classification problems when insufficient training information is available. In this investigation, a new semi-supervised classifier is proposed based on the concept of maximum vector-angular margin, (called S$^3$MAMC), the main goal of which is to find an optimal vector $c$ as close as possible to the center of the dataset consisting of both labeled samples and unlabeled samples. This makes S$^3$MAMC better generalization with smaller VC (Vapnik-Chervonenkis) dimension. However, S$^3$MAMC formulation is a non-convex model and therefore it is difficult to solve. Following that we present two optimization algorithms, mixed integer quadratic program (MIQP) and DC (difference of convex functions) program algorithms, to solve the S$^3$MAMC. Compared with the supervised learning methods, numerical experiments on real and synthetic databases demonstrate that the S$^3$MAMC can improve generalization when the labelled samples are relatively few. In addition, the S$^3$MAMC has competitive experiment results in generalization compared to the traditional semi-supervised classification methods.

A sparse logistic regression framework by difference of convex functions programming

Feature selection for logistic regression (LR) is still a challenging subject. In this paper, we present a new feature selection method for logistic regression based on a combination of the zero-norm and l2-norm regularization. However, discontinuity of the zero-norm makes it difficult to find the optimal solution. We apply a proper nonconvex approximation of the zero-norm to derive a robust difference of convex functions (DC) program. Moreover, DC optimization algorithm (DCA) is used to solve the problem effectively and the corresponding DCA converges linearly. Compared with traditional methods, numerical experiments on benchmark datasets show that the proposed method reduces the number of input features while maintaining accuracy. Furthermore, as a practical application, the proposed method is used to directly classify licorice seeds using near-infrared spectroscopy data. The simulation results in different spectral regions illustrates that the proposed method achieves equivalent classification performance to traditional logistic regressions yet suppresses more features. These results show the feasibility and effectiveness of the proposed method.

DC programming and DCA for sparse optimal scoring problem

Linear Discriminant Analysis (LDA) is a standard tool for classification and dimension reduction in many applications. However, the problem of high dimension is still a great challenge for the classical LDA. In this paper we consider the supervised pattern classification in the high dimensional setting, in which the number of features is much larger than the number of observations and present a novel approach to the sparse optimal scoring problem using the zero-norm. The difficulty in treating the zero-norm is overcome by using appropriate continuous approximations such that the resulting problems are solved by alternating schemes based on DC (Difference of Convex functions) programming and DCA (DC Algorithms). The experimental results on both simulated and real datasets show the efficiency of the proposed algorithms compared to the five state-of-the-art methods.

Secrecy Rate Maximization With Uncoordinated Cooperative Jamming by Single-Antenna Helpers Under Secrecy Outage Probability Constraint

The issue of physical layer security for single-input-multiple-output (SIMO) wiretap channel with uncoordinated cooperative jamming (UCJ) is addressed in this letter, and we focus on power allocation in secrecy rate maximization (SRM) problem. Differing from the existing works, a practical UCJ scheme is proposed by using multiple single-antenna helpers where each helper transmits a jamming signal independently to confound the eavesdropper. Assuming that statistical channel state information (CSI) concerning the eavesdropper is available, a modified DC (difference of convex function) programming method is provided to solve the SRM problem under secrecy outage probability constraint and an achievable solution of power allocation is obtained. Numerical results show that the proposed scheme has satisfactory secrecy performance especially when the number of helpers is large.

Sparse semi-supervised support vector machines by DC programming and DCA

This paper studies the problem of feature selection in the context of Semi-Supervised Support Vector Machine (S3VM). The zero norm, a natural concept dealing with sparsity, is used for feature selection purpose. Due to two nonconvex terms (the loss function of unlabeled data and the ℓ0 term), we are faced with a NP hard optimization problem. Two continuous approaches based on DC (Difference of Convex functions) programming and DCA (DC Algorithms) are developed. The first is DC approximation approach that approximates the ℓ0-norm by a DC function. The second is an exact reformulation approach based on exact penalty techniques in DC programming. All the resulting optimization problems are DC programs for which DCA are investigated. Several usual sparse inducing functions are considered, and six versions of DCA are developed. Empirical numerical experiments on several Benchmark datasets show the efficiency of the proposed algorithms, in both feature selection and classification.

DC approximation approaches for sparse optimization

Sparse optimization refers to an optimization problem involving the zero-norm in objective or constraints. In this paper, nonconvex approximation approaches for sparse optimization have been studied with a unifying point of view in DC (Difference of Convex functions) programming framework. Considering a common DC approximation of the zero-norm including all standard sparse inducing penalty functions, we studied the consistency between global minimums (resp. local minimums) of approximate and original problems. We showed that, in several cases, some global minimizers (resp. local minimizers) of the approximate problem are also those of the original problem. Using exact penalty techniques in DC programming, we proved stronger results for some particular approximations, namely, the approximate problem, with suitable parameters, is equivalent to the original problem. The efficiency of several sparse inducing penalty functions have been fully analyzed. Four DCA (DC Algorithm) schemes were developed that cover all standard algorithms in nonconvex sparse approximation approaches as special versions. They can be viewed as, an l1-perturbed algorithm/reweighted-l1 algorithm / reweighted-l2 algorithm. We offer a unifying nonconvex approximation approach, with solid theoretical tools as well as efficient algorithms based on DC programming and DCA, to tackle the zero-norm and sparse optimization. As an application, we implemented our methods for the feature selection in SVM (Support Vector Machine) problem and performed empirical comparative numerical experiments on the proposed algorithms with various approximation functions.

A DC programming approach for finding communities in networks.

Automatic discovery of community structures in complex networks is a fundamental task in many disciplines, including physics, biology, and the social sciences. The most used criterion for characterizing the existence of a community structure in a network is modularity, a quantitative measure proposed by Newman and Girvan (2004). The discovery community can be formulated as the so-called modularity maximization problem that consists of finding a partition of nodes of a network with the highest modularity. In this letter, we propose a fast and scalable algorithm called DCAM, based on DC (difference of convex function) programming and DCA (DC algorithms), an innovative approach in nonconvex programming framework for solving the modularity maximization problem. The special structure of the problem considered here has been well exploited to get an inexpensive DCA scheme that requires only a matrix-vector product at each iteration. Starting with a very large number of communities, DCAM furnishes, as output results, an optimal partition together with the optimal number of communities [Formula: see text]; that is, the number of communities is discovered automatically during DCAM's iterations. Numerical experiments are performed on a variety of real-world network data sets with up to 4,194,304 nodes and 30,359,198 edges. The comparative results with height reference algorithms show that the proposed approach outperforms them not only on quality and rapidity but also on scalability. Moreover, it realizes a very good trade-off between the quality of solutions and the run time.

A difference of convex functions algorithm for optimal scheduling and real-time assignment of preventive maintenance jobs on parallel processors

In this paper, we introduce a new approach based on DC (Difference of Convexfunctions) Programming and DCA (DC Algorithm) for minimizing the maintenancecost involving flow-time and tardiness penalties by optimal scheduling and real-timeassignment of preventive maintenance jobs on parallel processors. The main idea is to dividethe horizon considered into $H$ intervals. The problem is first formulatedas a mixed integer linear problem (MILP) and then reformulated as a DCprogram. A solution method based on DCA is used to solve the resultingproblem. The efficiency of DCA is compared with the algorithm based on thenew flow-time and tardiness rule (FTR) given in [1]. Thecomputational results on several test problems show that the solutionsprovided by DCA are better.

DCA based algorithms for multiple sequence alignment (MSA)

In the last years many techniques in bioinformatics have been developed for the central and complex problem of optimally aligning biological sequences. In this paper we propose a new optimization approach based on DC (Difference of Convex functions) programming and DC Algorithm (DCA) for the multiple sequence alignment in its equivalent binary linear program, called “Maximum Weight Trace” problem. This problem is beforehand recast as a polyhedral DC program with the help of exact penalty techniques in DC programming. Our customized DCA, requiring solution of a few linear programs, is original because it converges after finitely many iterations to a binary solution while it works in a continuous domain. To scale-up large-scale (MSA), a constraint generation technique is introduced in DCA. Preliminary computational experiments on benchmark data show the efficiency of the proposed algorithm DCAMSA, which generally outperforms some standard algorithms.

New and efficient DCA based algorithms for minimum sum-of-squares clustering

The purpose of this paper is to develop new efficient approaches based on DC (Difference of Convex functions) programming and DCA (DC Algorithm) to perform clustering via minimum sum-of-squares Euclidean distance. We consider the two most widely used models for the so-called Minimum Sum-of-Squares Clustering (MSSC in short) that are a bilevel programming problem and a mixed integer program. Firstly, the mixed integer formulation of MSSC is carefully studied and is reformulated as a continuous optimization problem via a new result on exact penalty technique in DC programming. DCA is then investigated to the resulting problem. Secondly, we introduce a Gaussian kernel version of the bilevel programming formulation of MSSC, named GKMSSC. The GKMSSC problem is formulated as a DC program for which a simple and efficient DCA scheme is developed. A regularization technique is investigated for exploiting the nice effect of DC decomposition and a simple procedure for finding good starting points of DCA is developed. The proposed DCA schemes are original and very inexpensive because they amount to computing, at each iteration, the projection of points onto a simplex and/or onto a ball, and/or onto a box, which are all determined in the explicit form. Numerical results on real word datasets show the efficiency, the scalability of DCA and its great superiority with respect to k-means and kernel k-means, standard methods for clustering.

A class of semi-supervised support vector machines by DC programming

This paper investigate a class of semi-supervised support vector machines ( $$\text{ S }^3\mathrm{VMs}$$ S 3 VMs ) with arbitrary norm. A general framework for the $$\text{ S }^3\mathrm{VMs}$$ S 3 VMs was first constructed based on a robust DC (Difference of Convex functions) program. With different DC decompositions, DC optimization formulations for the linear and nonlinear $$\text{ S }^3\mathrm{VMs}$$ S 3 VMs are investigated. The resulting DC optimization algorithms (DCA) only require solving simple linear program or convex quadratic program at each iteration, and converge to a critical point after a finite number of iterations. The effectiveness of proposed algorithms are demonstrated on some UCI databases and licorice seed near-infrared spectroscopy data. Moreover, numerical results show that the proposed algorithms offer competitive performances to the existing $$\text{ S }^3\mathrm{VM}$$ S 3 VM methods.

A class of smooth semi-supervised SVM by difference of convex functions programming and algorithm

Owing to its wide applicability, semi-supervised learning is an attractive method for using unlabeled data in classification. Applying a new smoothing strategy to a class of continuous semi-supervi...

A class of smooth semi-supervised SVM by difference of convex functions programming and algorithm

Owing to its wide applicability, semi-supervised learning is an attractive method for using unlabeled data in classification. Applying a new smoothing strategy to a class of continuous semi-supervised support vector machines (S3VMs), this paper proposes a class of smooth S3VMs (S4VMs) without adding new variables and constraints to the corresponding S3VMs. Moreover, a general framework for solving the S4VMs is constructed based on robust DC (difference of convex functions) programming. Furthermore, DC optimization algorithms (DCAs) for solving the S4VMs are investigated. The resulting DCAs converge and only require solving one linear or quadratic program at each iteration. Numerical experiments on some real-world databases demonstrate that the proposed smooth S3VMs are feasible and effective, and have comparable results as other S3VMs.

Long-Short Portfolio Optimization Under Cardinality Constraints by Difference of Convex Functions Algorithm

In the matter of Portfolio selection, we consider an extended version of the Mean-Absolute Deviation (MAD) model, which includes discrete asset choice constraints (threshold and cardinality constraints) and one is allowed to sell assets short if it leads to a better risk-return tradeoff. Cardinality constraints limit the number of assets in the optimal portfolio and threshold constraints limit the amount of capital to be invested in (or sold short from) each asset and prevent very small investments in (or short selling from) any asset. The problem is formulated as a mixed 0---1 programming problem, which is known to be NP-hard. Attempting to use DC (Difference of Convex functions) programming and DCA (DC Algorithms), an efficient approach in non-convex programming framework, we reformulate the problem in terms of a DC program, and investigate a DCA scheme to solve it. Some computational results carried out on benchmark data sets show that DCA has a better performance in comparison to the standard solver IBM CPLEX.

Recognition of the hardness of licorice seeds using a semi-supervised learning method and near-infrared spectral data

The recognition of the hardness of licorice seeds is a challenging task. The purpose of this investigation is to identify the hardness of licorice seeds employing a semi-supervised learning method and near-infrared spectroscopy. An excellent semi-supervised learning model, the semi-supervised support vector machine (S3VM), is built using the small labeled samples and the large unlabeled samples. Moreover, the proposed model is solved by employing an effective method, the robust DC (difference of convex functions) programming. The resulting algorithm only requires the solving of a few linear programs. Furthermore, this model is used for the direct classification of licorice samples. Comparing with the supervised support vector machine (SVM), experimental results on different spectral regions show that incorporating unlabeled samples in training improves the generalization when insufficient training information is available. Moreover, our method outperforms the existing S3VM method by obtaining better performance in different spectral regions. These results show that it is possible to identify the hardness of licorice seeds using the proposed S3VM and near-infrared spectroscopic data. We hope that the results obtained in this study will help further investigations of the hardness of crop seeds.

A multiple kernel framework for inductive semi-supervised SVM learning

We investigate the benefit of combining both cluster assumption and manifold assumption underlying most of the semi-supervised algorithms using the flexibility and the efficiency of multiple kernel learning. The multiple kernel version of Transductive SVM (a cluster assumption based approach) is proposed and it is solved based on DC (Difference of Convex functions) programming. Promising results on benchmark data sets and the BCI data analysis suggest and support the effectiveness of proposed work.

Binary classification via spherical separator by DC programming and DCA

In this paper, we consider a binary supervised classification problem, called spherical separation, that consists of finding, in the input space or in the feature space, a minimal volume sphere separating the set $${\mathcal{A}}$$ from the set $${\mathcal{B}}$$ (i.e. a sphere enclosing all points of $${ \mathcal{A}}$$ and no points of $${\mathcal{B}}$$ ). The problem can be cast into the DC (Difference of Convex functions) programming framework and solved by DCA (DC Algorithm) as shown in the works of Astorino et al. (J Glob Optim 48(4):657---669, 2010). The aim of this paper is to investigate more attractive DCA based algorithms for this problem. We consider a new optimization model and propose two interesting DCA schemes. In the first scheme we have to solve a quadratic program at each iteration, while in the second one all calculations are explicit. Numerical simulations show the efficiency of our customized DCA with respect to the methods developed in Astorino et al.