Difference Of Convex Functions Programming Research Articles

Markov chain stochastic DCA and applications in deep learning with PDEs regularization

This paper addresses a large class of nonsmooth nonconvex stochastic DC (difference-of-convex functions) programs where endogenous uncertainty is involved and i.i.d. (independent and identically distributed) samples are not available. Instead, we assume that it is only possible to access Markov chains whose sequences of distributions converge to the target distributions. This setting is legitimate as Markovian noise arises in many contexts including Bayesian inference, reinforcement learning, and stochastic optimization in high-dimensional or combinatorial spaces. We then design a stochastic algorithm named Markov chain stochastic DCA (MCSDCA) based on DCA (DC algorithm) - a well-known method for nonconvex optimization. We establish the convergence analysis in both asymptotic and nonasymptotic senses. The MCSDCA is then applied to deep learning via PDEs (partial differential equations) regularization, where two realizations of MCSDCA are constructed, namely MCSDCA-odLD and MCSDCA-udLD, based on overdamped and underdamped Langevin dynamics, respectively. Numerical experiments on time series prediction and image classification problems with a variety of neural network topologies show the merits of the proposed methods.

A generalized adaptive robust distance metric driven smooth regularization learning framework for pattern recognition

In this work, we propose a novel generalized adaptive robust distance metric called Lδ(u). Compared with other distance metrics, Lδ(u) has some desirable salient properties, such as symmetry, boundedness, robustness, nonconvexity, and adaptivity, with both first-order and higher-order moments from samples. On the other hand, Lδ(u) can pick different robust distance metrics for different learning tasks during the learning process by the adaptive parameter δ. Furthermore, we apply Lδ(u) to twin extreme learning machine (TELM) and develop an smooth regularized TELM learning framework for supervised and semi-supervised classification. By introducing the structural risk minimization (SRM) principle and smoothing techniques, the learning framework perfectly overcomes the computational burden associated with the matrix inversion operation required during TELM solving, while also significantly improving performance. More importantly, the proposed learning framework not only improves the robustness of TELM, but also can effectively use the geometric information of the marginal distribution embedded in the unlabeled samples to construct a more reasonable classifier. Finally, the globally convergent and quadratic convergent fast Newton-Armijo algorithm and DC (difference of convex functions) programming algorithm (DCA) are designed to solve the proposed methods. Experimental results demonstrate the effectiveness and robustness of our methods.

Impulse-to-Peak Optimization of Positive Linear Systems via DC Programming

This paper is concerned with the optimization of the impulse-to-peak value of positive linear systems. Although the results of the analysis of the peak value of positive linear systems have been reported in the literature, the synthesis problem of impulse-to-peak value remains unresolved. To address this problem, in this paper we show that a cost-constrained minimization problem of the impulse-to-peak value of parameterized positive linear systems can be reduced to a DC (difference of convex functions) program. The derivation utilizes the log-log convexity of posynomial functions and a recent characterization of the impulse-to-peak value of positive systems via a linear program. To illustrate the effectiveness of our results, we study the optimization problem for minimizing the peak infection of networked epidemic spreading processes.

Dynamic and Robust Sensor Selection Strategies for Wireless Positioning With TOA/RSS Measurement

Emerging wireless applications are requiring ever more accurate location-positioning from sensor measurements. In this paper, we develop sensor selection strategies for 3D wireless positioning based on time of arrival (TOA) and received signal strength (RSS) measurements to handle two distinct scenarios: (i) known approximated target location, for which we conduct dynamic sensor selection to minimize the positioning error; and (ii) unknown approximated target location, in which the worst-case positioning error is minimized via robust sensor selection. We derive expressions for the Cramér-Rao lower bound (CRLB) as a performance metric to quantify the positioning accuracy resulted from selected sensors. For dynamic sensor selection, two greedy selection strategies are proposed, each of which exploits properties revealed in the derived CRLB expressions. These selection strategies are shown to strike an efficient balance between computational complexity and performance suboptimality. For robust sensor selection, we show that the conventional convex relaxation approach leads to instability, and then develop three algorithms based on (i) iterative convex optimization (ICO), (ii) difference of convex functions programming (DCP), and (iii) discrete monotonic optimization (DMO). Each of these strategies exhibits a different tradeoff between computational complexity and optimality guarantee. Simulation results show that the proposed sensor selection strategies provide significant improvements in terms of accuracy and/or complexity compared to existing sensor selection methods.

Stochastic Difference-of-Convex-Functions Algorithms for Nonconvex Programming

The paper deals with stochastic difference-of-convex-functions (DC) programs, that is, optimization problems whose cost function is a sum of a lower semicontinuous DC function and the expectation of a stochastic DC function with respect to a probability distribution. This class of nonsmooth and nonconvex stochastic optimization problems plays a central role in many practical applications. Although there are many contributions in the context of convex and/or smooth stochastic optimization, algorithms dealing with nonconvex and nonsmooth programs remain rare. In deterministic optimization literature, the DC algorithm (DCA) is recognized to be one of the few algorithms able to effectively solve nonconvex and nonsmooth optimization problems. The main purpose of this paper is to present some new stochastic DCAs for solving stochastic DC programs. The convergence analysis of the proposed algorithms is carefully studied, and numerical experiments are conducted to justify the algorithms' behaviors.

DCA for Sparse Quadratic Kernel-Free Least Squares Semi-Supervised Support Vector Machine

With the development of science and technology, more and more data have been produced. For many of these datasets, only some of the data have labels. In order to make full use of the information in these data, it is necessary to classify them. In this paper, we propose a strong sparse quadratic kernel-free least squares semi-supervised support vector machine (SSQLSS3VM), in which we add a ℓ0norm regularization term to make it sparse. An NP-hard problem arises since the proposed model contains the ℓ0 norm and another nonconvex term. One important method for solving the nonconvex problem is the DC (difference of convex function) programming. Therefore, we first approximate the ℓ0 norm by a polyhedral DC function. Moreover, due to the existence of the nonsmooth terms, we use the sGS-ADMM to solve the subproblem. Finally, empirical numerical experiments show the efficiency of the proposed algorithm.

Resource Allocation for D2D Cellular Networks With QoS Constraints: A DC Programming- Based Approach

Device-to-device (D2D) communications provide efficient ways to increase spectrum utilization ratio with reduced power consumption for proximity wireless applications. In this paper, we investigate resource allocation strategies for D2D communications underlaying cellular networks. To be specific, we study the centralized resource allocation algorithm for controlling transmit powers of the underlying D2D pairs in order to maximize the weighted sum-rate while guaranteeing the quality of service (QoS) requirements for both D2D pairs and cellular users (CUs). A novel DC (difference of convex function) programming-based method, called alternative DC (ADC) programming, is introduced to effectively solve this complicated resource allocation problem. Through updating each D2D pair’s power alternatively, the QoS requirement for each D2D pair can be solvable and incorporated systematically to the introduced ADC programming framework. The simulation results show that the introduced ADC programming achieves the highest weighted sum-rate compared to the state-of-the-art methods while ensuring that the QoS of each D2D pair and CU are satisfied.

DCA based approaches for bi-level variable selection and application for estimate multiple sparse covariance matrices

Variable selection plays an important role in analyzing high dimensional data and is a fundamental problem in machine learning. When the data possesses certain group structures in which individual variables are also meaningful scientifically, we are naturally interested in selecting important groups as well as important variables within the selected groups. This is referred to as the bi-level variable selection which is much more complex than the selection of individual variables. In recent years, research on the topic of variable selection is very active, but the majority of the work is focused on the individual variable selection. There is therefore a need to further develop more effective approaches for bi-level variable selection. Since DC (Difference of Convex functions) programming and DCA (DC Algorithm), powerful tools in nonconvex programming framework, have been successfully investigated for individual variable selection, we believe that they could be efficiently exploited for the more difficult bi-level variable selection task. In that direction, we investigate in this work DC approximations of the mixed zero norm (ℓ0,0) and the combined norm (ℓ0+ℓq,0). The resulting approximate problems are then formulated as DC programs for which DCA based algorithms are introduced. As an application, these DCA schemes are developed for estimating multiple sparse covariance matrices sharing some common structures such as the locations or weights of non-zero elements. The experimental results on both simulated and real datasets indicate the efficiency of our algorithms.

Refined Nonlinear Rectenna Modeling and Optimal Waveform Design for Multi-User Multi-Antenna Wireless Power Transfer

In this paper, we study the optimal waveform design for wireless power transfer (WPT) from a multi-antenna energy transmitter (ET) to multiple single-antenna energy receivers (ERs) simultaneously in multi-path frequency-selective channels. First, we propose a refined nonlinear current-voltage model of the diode in the ER rectifier, and accordingly derive new expressions for the output direct current (DC) voltage and corresponding harvested power at the ER. Leveraging this new rectenna model, we first consider the single-ER case and study the multisine-based power waveform design based on the wireless channel to maximize the harvested power at the ER. We propose two efficient algorithms for finding high-quality suboptimal solutions to this non-convex optimization problem. Next, we extend our formulated waveform design problem to the general multi-ER case for maximizing the weighted sum of the harvested powers by all ERs, and propose an efficient difference-of-convex-functions programming (DCP)-based algorithm for solving this problem. Finally, we demonstrate the superior performance of our proposed waveform designs based on the new rectenna model over existing schemes/models via simulations.

Alternating DCA for reduced-rank multitask linear regression with covariance matrix estimation

We study a challenging problem in machine learning that is the reduced-rank multitask linear regression with covariance matrix estimation. The objective is to build a linear relationship between multiple output variables and input variables of a multitask learning process, taking into account the general covariance structure for the errors of the regression model in one hand, and reduced-rank regression model in another hand. The problem is formulated as minimizing a nonconvex function in two joint matrix variables (X,Θ) under the low-rank constraint on X and positive definiteness constraint on Θ. It has a double difficulty due to the non-convexity of the objective function as well as the low-rank constraint. We investigate a nonconvex, nonsmooth optimization approach based on DC (Difference of Convex functions) programming and DCA (DC Algorithm) for this hard problem. A penalty reformulation is considered which takes the form of a partial DC program. An alternating DCA and its inexact version are developed, both algorithms converge to a weak critical point of the considered problem. Numerical experiments are performed on several synthetic and benchmark real multitask linear regression datasets. The numerical results show the performance of the proposed algorithms and their superiority compared with three classical alternating/joint methods.

DCA for online prediction with expert advice

We investigate DC (Difference of Convex functions) programming and DCA (DC Algorithm) for a class of online learning techniques, namely prediction with expert advice, where the learner’s prediction is made based on the weighted average of experts’ predictions. The problem of predicting the experts’ weights is formulated as a DC program for which an online version of DCA is investigated. The two so-called approximate/complete variants of online DCA based schemes are designed, and their regrets are proved to be logarithmic/sublinear. The four proposed algorithms for online prediction with expert advice are furthermore applied to online binary classification. Experimental results tested on various benchmark datasets showed their performance and their superiority over three standard online prediction with expert advice algorithms—the well-known weighted majority algorithm and two online convex optimization algorithms.

DCA-based algorithms for DC fitting

We investigate a nonconvex, nonsmooth optimization approach based on DC (Difference of Convex functions) programming and DCA (DC Algorithm) for the so-called DC fitting problem, which aims to fit a given set of data points by a DC function. The problem is tackled as minimizing the squared Euclidean norm fitting error. It is formulated as a DC program for which a standard DCA scheme is developed. Furthermore, a modified DCA scheme with successive DC decomposition is proposed. These standard/modified versions of DCA are applied for solving the continuous piecewise-linear fitting problem. Numerical experiments on many synthetic and real datasets with small-to-large sizes show the efficiency of our DCA-based approach in comparison with the existing approaches for constructing continuous piecewise-linear models.

Community detection via an efficient nonconvex optimization approach based on modularity

Maximizing modularity is a widely used method for community detection, which is generally solved by approximate or greedy search because of its high complexity. In this paper, we propose a method, named MSM, for modularity maximization, which reformulates the modularity maximization problem as a subset identification problem and maximizes the surrogate of the modularity. The surrogate of the modularity is constructed by replacing the discontinuous indicator functions in the reformulated modularity function with the continuous truncated L1 function. This makes the NP-hard problem of maximizing the modularity function approximately become a non-convex optimization problem, which can be efficiently solved via the DC (Difference of Convex Functions) Programming. The proposed MSM method can be used for community detection when the number of communities is given, and it can also be applied to the situation where the number of communities is unknown. Then, we demonstrate the advantages of the proposed MSM method by some simulation results and real data analyses.

Novel DCA based algorithms for a special class of nonconvex problems with application in machine learning

We address the problem of minimizing the sum of a nonconvex, differentiable function and composite functions by DC (Difference of Convex functions) programming and DCA (DC Algorithm), powerful tools of nonconvex optimization. The main idea of DCA relies on DC decompositions of the objective function, it consists in approximating a DC (nonconvex) program by a sequence of convex ones. We first develop a standard DCA scheme especially dealing with the very specific structure of this problem. Furthermore, we extend DCA to give rise to the so-named DCA-Like, which is based on a new and efficient way to approximate the DC objective function without knowing a DC decomposition. We further improve DCA based algorithms by incorporating the Nesterov’s acceleration technique into them. The convergence properties and the convergence rate under Kurdyka-Łojasiewicz assumption of extended DCAs are rigorously studied. We prove that DCA-Like and the accelerated versions subsequently converge from every initial point to a critical point of the considered problem. Finally, we investigate the proposed algorithms for an important problem in machine learning: the t-distributed stochastic neighbor embedding. Numerical experiments on several benchmark datasets illustrate the efficiency of our algorithms.

Adaptive robust learning framework for twin support vector machine classification

In general, introducing robust distance metrics and loss functions in the learning process can improve the robustness of the algorithms. In this work, we first propose a new robust loss function called adaptive capped Lθε-loss. For different problems, we can choose different loss functions through adaptive parameter θ during the learning process. Secondly, we propose a new robust distance metric induced by correntropy (CIM) that is based on Laplacian kernel. The CIM contains first and higher-order moments from samples. Further, we demonstrate some important and interesting properties of the Lθε-loss and CIM, such as robustness, boundedness, nonconvexity, etc. Finally, we apply the to Lθε-loss and CIM to twin support vector machine (TWSVM) and develop an adaptive robust learning framework, namely adaptive robust twin support vector machine (ARTSVM). The proposed ARTSVM not only inherits the advantages of TWSVM but also improves the robustness of classification problems. A non-convex optimization method, DC (difference of convex functions) programming algorithm (DCA) is used to solve the proposed ARTSVM, and the convergence of the algorithm is proved theoretically. Experiments on multiple datasets show that the proposed ARTSVM is competitive with existing methods.

Robust sparse principal component analysis by DC programming algorithm

The classical principal component analysis (PCA) is not sparse enough since it is based on the L2-norm that is also prone to be adversely affected by the presence of outliers and noises. In order to address the problem, a sparse robust PCA framework is proposed based on the min of zero-norm regularization and the max of Lp-norm (0 < p ≤ 2) PCA. Furthermore, we developed a continuous optimization method, DC (difference of convex functions) programming algorithm (DCA), to solve the proposed problem. The resulting algorithm (called DC-LpZSPCA) is convergent linearly. In addition, when choosing different p values, the model can keep robust and is applicable to different data types. Numerical simulations are simulated in artificial data sets and Yale face data sets. Experiment results show that the proposed method can maintain good sparsity and anti-outlier ability.

MM algorithms for distance covariance based sufficient dimension reduction and sufficient variable selection

Sufficient dimension reduction (SDR) using distance covariance (DCOV) was recently proposed as an approach to dimension-reduction problems. Compared with other SDR methods, it is model-free without estimating link function and does not require any particular distributions on predictors. However, the DCOV-based SDR method involves optimizing a nonsmooth and nonconvex objective function over the Stiefel manifold. To tackle the numerical challenge, the original objective function is equivalently formulated into a DC (Difference of Convex functions) program and an iterative algorithm based on the majorization–minimization (MM) principle is constructed. At each step of the MM algorithm, one iteration of Riemannian Newton’s method is taken to solve the quadratic subproblem on the Stiefel manifold inexactly. In addition, the algorithm can also be readily extended to sufficient variable selection (SVS) using distance covariance. Finally, the convergence property of the proposed algorithm under some regularity conditions is established. Simulation and real data analysis show our algorithm drastically improves the computation efficiency and is robust across various settings compared with the existing method. Matlab codes implementing our methods and scripts for regenerating the numerical results are available at https://github.com/runxiong-wu/MMRN.

Federated Learning via Over-the-Air Computation

The stringent requirements for low-latency and privacy of the emerging high-stake applications with intelligent devices such as drones and smart vehicles make the cloud computing inapplicable in these scenarios. Instead, edge machine learning becomes increasingly attractive for performing training and inference directly at network edges without sending data to a centralized data center. This stimulates a nascent field termed as federated learning for training a machine learning model on computation, storage, energy and bandwidth limited mobile devices in a distributed manner. To preserve data privacy and address the issues of unbalanced and non-IID data points across different devices, the federated averaging algorithm has been proposed for global model aggregation by computing the weighted average of locally updated model at each selected device. However, the limited communication bandwidth becomes the main bottleneck for aggregating the locally computed updates. We thus propose a novel over-the-air computation based approach for fast global model aggregation via exploring the superposition property of a wireless multiple-access channel. This is achieved by joint device selection and beamforming design, which is modeled as a sparse and low-rank optimization problem to support efficient algorithms design. To achieve this goal, we provide a difference-of-convex-functions (DC) representation for the sparse and low-rank function to enhance sparsity and accurately detect the fixed-rank constraint in the procedure of device selection. A DC algorithm is further developed to solve the resulting DC program with global convergence guarantees. The algorithmic advantages and admirable performance of the proposed methodologies are demonstrated through extensive numerical results.

A new multiple kernel-based regularization method for identification of delay linear dynamic systems

Model parameter estimation, model order selection, variable selection and time delay estimation are four important issues that receive increasing interests in dynamic system identification. However, previous work did not solve the four issues simultaneously. Motivated by the multiple kernel-based regularization method (MKRM), this paper proposes a new multiple kernel-based regularization method for joint model parameter estimation, model order selection, variable selection and time delay estimation for delay linear dynamic systems, referred to as the MKRM-D. Then, an efficient iterative reweighted algorithm is derived to solve the resulting difference of convex functions programming (DCP) problem. In addition, by exploiting the structure of the objective function in each iteration of this algorithm, the alternating direction method of multipliers (ADMM) is employed to decompose the centralized problem into a series of independent subproblems with lower variable dimension, which can be solved in a parallel and distributed manner. The performance of the proposed method is demonstrated by numerical experiments using both synthetic and real data.

Scheduling of steelmaking-continuous casting process using deflected surrogate Lagrangian relaxation approach and DC algorithm

This paper investigates a hybrid flowshop scheduling (HFS) problem in the steelmaking continuous casting (SCC) process. Firstly, a mathematical model is built for the SCC scheduling problem. By relaxing the machine capacity constraint, the SCC scheduling problem can be transformed into a DC (difference of convex functions) programming problem, which can solved by using DC algorithm. Under some reasonable assumptions, the convergence of the DC algorithm is analyzed. Secondly, we propose an effective and efficient deflected surrogate subgradient method with global convergence to solve the Lagrangian dual (LD) problem. Thirdly, a simple heuristic method is designed to obtain a feasible scheduling. Lastly, we report some computational experiments to demonstrate the effectiveness of the proposed surrogate subgradient method by comparing with other similar surrogate subgradient methods.