Continuous Relaxations Research Articles

Exact and Approximation Algorithms for Sparse Principal Component Analysis

Sparse principal component analysis (SPCA) is designed to enhance the interpretability of traditional principal component analysis by optimally selecting a subset of features that comprise the first principal component. Given the NP-hard nature of SPCA, most current approaches resort to approximate solutions, typically achieved through tractable semidefinite programs or heuristic methods. To solve SPCA to optimality, we propose two exact mixed-integer semidefinite programs (MISDPs) and an arbitrarily equivalent mixed-integer linear program. The MISDPs allow us to design an effective branch-and-cut algorithm with closed-form cuts that do not need to solve dual problems. For the proposed mixed-integer formulations, we further derive the theoretical optimality gaps of their continuous relaxations. Besides, we apply the greedy and local search algorithms to solving SPCA and derive their first-known approximation ratios. Our numerical experiments reveal that the exact methods we developed can efficiently find optimal solutions for data sets containing hundreds of features. Furthermore, our approximation algorithms demonstrate both scalability and near-optimal performance when benchmarked on larger data sets, specifically those with thousands of features. History: Accepted by Andrea Lodi, Area Editor for Design & Analysis of Algorithms—Discrete. Funding: This research was supported in part by the Division of Civil, Mechanical and Manufacturing Innovation [Grant 224614], the Division of Computing and Communication Foundations [Grant 2246417], and the Office of Naval Research [Grant N00014-24-1-2066]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.0372 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0372 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Code and Data Repository for Exact and Approximation Algorithms for Sparse Principal Component Analysis

Sparse principal component analysis (SPCA) is designed to enhance the interpretability of traditional principal component analysis by optimally selecting a subset of features that comprise the first principal component. Given the NP-hard nature of SPCA, most current approaches resort to approximate solutions, typically achieved through tractable semidefinite programs or heuristic methods. To solve SPCA to optimality, we propose two exact mixed-integer semidefinite programs (MISDPs) and an arbitrarily equivalent mixed-integer linear program. The MISDPs allow us to design an effective branch-and-cut algorithm with closed-form cuts that do not need to solve dual problems. For the proposed mixed-integer formulations, we further derive the theoretical optimality gaps of their continuous relaxations. Besides, we apply the greedy and local search algorithms to solving SPCA and derive their first-known approximation ratios. Our numerical experiments reveal that the exact methods we developed can efficiently find optimal solutions for data sets containing hundreds of features. Furthermore, our approximation algorithms demonstrate both scalability and near-optimal performance when benchmarked on larger data sets, specifically those with thousands of features.

Uncertainty reduction in robust optimization

Uncertainty reduction has recently been introduced in the robust optimization literature as a relevant special case of decision-dependent uncertainty. Herein, we identify two relevant situations in which the problem is polynomially solvable. We further provide insights into possible MILP reformulations and the strength of their continuous relaxations.

Text Feature Adversarial Learning for Text Generation With Knowledge Transfer From GPT2.

Text generation is a key component of many natural language tasks. Motivated by the success of generative adversarial networks (GANs) for image generation, many text-specific GANs have been proposed. However, due to the discrete nature of text, these text GANs often use reinforcement learning (RL) or continuous relaxations to calculate gradients during learning, leading to high-variance or biased estimation. Furthermore, the existing text GANs often suffer from mode collapse (i.e., they have limited generative diversity). To tackle these problems, we propose a new text GAN model named text feature GAN (TFGAN), where adversarial learning is performed in a continuous text feature space. In the adversarial game, GPT2 provides the "true" features, while the generator of TFGAN learns from them. TFGAN is trained by maximum likelihood estimation on text space and adversarial learning on text feature space, effectively combining them into a single objective, while alleviating mode collapse. TFGAN achieves appealing performance in text generation tasks, and it can also be used as a flexible framework for learning text representations.

InitialGAN: A Language GAN With Completely Random Initialization.

Text generative models trained via maximum likelihood estimation (MLE) suffer from the notorious exposure bias problem, and generative adversarial networks (GANs) are shown to have potential to tackle this problem. The existing language GANs adopt estimators, such as REINFORCE or continuous relaxations to model word probabilities. The inherent limitations of such estimators lead current models to rely on pretraining techniques (MLE pretraining or pretrained embeddings). Representation modeling methods (RMMs), which are free from those limitations, however, are seldomly explored because of their poor performance in previous attempts. Our analyses reveal that invalid sampling methods and unhealthy gradients are the main contributors to such unsatisfactory performance. In this work, we present two techniques to tackle these problems: dropout sampling and fully normalized long short-term memory network (LSTM). Based on these two techniques, we propose InitialGAN whose parameters are randomly initialized in full. Besides, we introduce a new evaluation metric, least coverage rate (LCR), to better evaluate the quality of generated samples. The experimental results demonstrate that the InitialGAN outperforms both MLE and other compared models. To the best of our knowledge, it is the first time a language GAN can outperform MLE without using any pretraining techniques.

New optimization models for optimal classification trees

The most efficient state-of-the-art methods to build classification trees are greedy heuristics (e.g. CART) that may fail to find underlying characteristics in datasets. Recently, exact linear formulations that have shown better accuracy were introduced. However they do not scale up to datasets with more than a few thousands data points. In this paper, we introduce four new formulations for building optimal trees. The first one is a quadratic model based on the well-known formulation of Bertsimas et al. We then propose two different linearizations of this new quadratic model. The last model is an extension to real-valued datasets of a flow-formulation limited to binary datasets (Aghaei et al.). Each model is introduced for both axis-aligned and oblique splits. We further prove that our new formulations have stronger continuous relaxations than existing models. Finally, we present computational results on 22 standard datasets with up to thousands of data points. Our exact models have reduced solution times with learning performances as strong or significantly better than state of the art exact approaches.

Profit-based unit commitment models with price-responsive decision-dependent uncertainty

Highlighting the increasing importance of demand elasticity in electricity markets and its impact on the revenues of power generating companies, this paper proposes new profit-based unit commitment models that effectively capture the uncertainty in the willingness-to-pay the price set for the elastic demand. To develop a new revenue scheme for power generating companies, we use a coupling function to model the willingness-to-pay response of the elastic demand as a decision-dependent source of uncertainty. The coupling function reflects how power generating companies’ pricing decisions may influence the market appeal (i.e., the buyer’s willingness-to-pay) and how it affects their revenues. The optimization models are stochastic mixed-integer nonlinear problems with nonconvex continuous relaxations and are not amenable to a numerical solution in their original forms. We devise a convexification reformulation method and derive valid inequalities to strengthen the formulation. We propose a learning framework to parameterize the willingness-to-pay functions and the concept of the value of the decision-dependent solution to quantify the value of the uncertainty modeling approach. Numerical tests on power systems of various sizes, demand portfolios, and price elasticity levels show (i) how the valid inequalities speed up the solution process, (ii) the benefits of properly modeling decision-dependent uncertainty and demand elasticity, and (iii) how the incorporation of decision-dependent uncertainty in demand elasticity can change the power generating companies’ decisions and revenue estimation.

Beyond ℓ1 sparse coding in V1.

Growing evidence indicates that only a sparse subset from a pool of sensory neurons is active for the encoding of visual stimuli at any instant in time. Traditionally, to replicate such biological sparsity, generative models have been using the ℓ1 norm as a penalty due to its convexity, which makes it amenable to fast and simple algorithmic solvers. In this work, we use biological vision as a test-bed and show that the soft thresholding operation associated to the use of the ℓ1 norm is highly suboptimal compared to other functions suited to approximating ℓp with 0 ≤ p < 1 (including recently proposed continuous exact relaxations), in terms of performance. We show that ℓ1 sparsity employs a pool with more neurons, i.e. has a higher degree of overcompleteness, in order to maintain the same reconstruction error as the other methods considered. More specifically, at the same sparsity level, the thresholding algorithm using the ℓ1 norm as a penalty requires a dictionary of ten times more units compared to the proposed approach, where a non-convex continuous relaxation of the ℓ0 pseudo-norm is used, to reconstruct the external stimulus equally well. At a fixed sparsity level, both ℓ0- and ℓ1-based regularization develop units with receptive field (RF) shapes similar to biological neurons in V1 (and a subset of neurons in V2), but ℓ0-based regularization shows approximately five times better reconstruction of the stimulus. Our results in conjunction with recent metabolic findings indicate that for V1 to operate efficiently it should follow a coding regime which uses a regularization that is closer to the ℓ0 pseudo-norm rather than the ℓ1 one, and suggests a similar mode of operation for the sensory cortex in general.

A Solver for Multiobjective Mixed-Integer Convex and Nonconvex Optimization

AbstractThis paper proposes a general framework for solving multiobjective nonconvex optimization problems, i.e., optimization problems in which multiple objective functions have to be optimized simultaneously. Thereby, the nonconvexity might come from the objective or constraint functions, or from integrality conditions for some of the variables. In particular, multiobjective mixed-integer convex and nonconvex optimization problems are covered and form the motivation of our studies. The presented algorithm is based on a branch-and-bound method in the pre-image space, a technique which was already successfully applied for continuous nonconvex multiobjective optimization. However, extending this method to the mixed-integer setting is not straightforward, in particular with regard to convergence results. More precisely, new branching rules and lower bounding procedures are needed to obtain an algorithm that is practically applicable and convergent for multiobjective mixed-integer optimization problems. Corresponding results are a main contribution of this paper. What is more, for improving the performance of this new branch-and-bound method we enhance it with two types of cuts in the image space which are based on ideas from multiobjective mixed-integer convex optimization. Those combine continuous convex relaxations with adaptive cuts for the convex hull of the mixed-integer image set, derived from supporting hyperplanes to the relaxed sets. Based on the above ingredients, the paper provides a new multiobjective mixed-integer solver for convex problems with a stopping criterion purely in the image space. What is more, for the first time a solver for multiobjective mixed-integer nonconvex optimization is presented. We provide the results of numerical tests for the new algorithm. Where possible, we compare it with existing procedures.

Binary programs for asymmetric betweenness problems and relations to the quadratic linear ordering problem

We present and compare novel binary programs for linear ordering problems that involve the notion of asymmetric betweenness and expose relations to the quadratic linear ordering problem and its linearization. While two of the binary programs prove particularly superior from a computational point of view when many or all betweenness relations shall be modeled, the others arise as natural formulations that resemble important theoretical correspondences and provide a compact alternative for sparse problem instances. A reasoning for the strengths and weaknesses of the different formulations is derived by means of polyhedral considerations with respect to their continuous relaxations.

On piecewise linear approximations of bilinear terms: structural comparison of univariate and bivariate mixed-integer programming formulations

Bilinear terms naturally appear in many optimization problems. Their inherent non-convexity typically makes them challenging to solve. One approach to tackle this difficulty is to use bivariate piecewise linear approximations for each variable product, which can be represented via mixed-integer linear programming (MIP) formulations. Alternatively, one can reformulate the variable products as a sum of univariate functions. Each univariate function can again be approximated by a piecewise linear function and modelled via an MIP formulation. In the literature, heterogeneous results are reported concerning which approach works better in practice, but little theoretical analysis is provided. We fill this gap by structurally comparing bivariate and univariate approximations with respect to two criteria. First, we compare the number of simplices sufficient for an varepsilon -approximation. We derive upper bounds for univariate approximations and compare them to a lower bound for bivariate approximations. We prove that for a small prescribed approximation error varepsilon , univariate varepsilon -approximations require fewer simplices than bivariate varepsilon -approximations. The second criterion is the tightness of the continuous relaxations (CR) of corresponding sharp MIP formulations. Here, we prove that the CR of a bivariate MIP formulation describes the convex hull of a variable product, the so-called McCormick relaxation. In contrast, we show by a volume argument that the CRs corresponding to univariate approximations are strictly looser. This allows us to explain many of the computational effects observed in the literature and to give theoretical evidence on when to use which kind of approximation.

Dantzig–Wolfe reformulations for binary quadratic problems

The purpose of this paper is to provide strong reformulations for binary quadratic problems. We propose a first methodological analysis on a family of reformulations combining Dantzig–Wolfe and Quadratic Convex optimization principles. We show that a few reformulations of our family yield continuous relaxations that are strong in terms of dual bounds and computationally efficient to optimize. As a representative case study, we apply them to a cardinality constrained quadratic knapsack problem, providing extensive experimental insights. We report and analyze in depth a particular reformulation providing continuous relaxations whose solutions turn out to be integer optima in all our tests.

Gaining or losing perspective

We study MINLO (mixed-integer nonlinear optimization) formulations of the disjunction $$x\in \{0\}\cup [l,u]$$ , where z is a binary indicator of $$x\in [l,u]$$ ( $$u> \ell > 0$$ ), and y “captures” f(x), which is assumed to be convex on its domain [l, u], but otherwise $$y=0$$ when $$x=0$$ . This model is useful when activities have operating ranges, we pay a fixed cost for carrying out each activity, and costs on the levels of activities are convex. Using volume as a measure to compare convex bodies, we investigate a variety of continuous relaxations of this model, one of which is the convex-hull, achieved via the “perspective reformulation” inequality $$y \ge zf(x/z)$$ . We compare this to various weaker relaxations, studying when they may be considered as viable alternatives. In the important special case when $$f(x) := x^p$$ , for $$p>1$$ , relaxations utilizing the inequality $$yz^q \ge x^p$$ , for $$q \in [0,p-1]$$ , are higher-dimensional power-cone representable, and hence tractable in theory. One well-known concrete application (with $$f(x) := x^2$$ ) is mean-variance optimization (in the style of Markowitz), and we carry out some experiments to illustrate our theory on this application.

Low-Depth Mechanisms for Quantum Optimization

One of the major application areas of interest for both near-term and fault-tolerant quantum computers is the optimization of classical objective functions. In this work, we develop intuitive constructions for a large class of these algorithms based on connections to simple dynamics of quantum systems, quantum walks, and classical continuous relaxations. We focus on developing a language and tools connected with kinetic energy on a graph for understanding the physical mechanisms of success and failure to guide algorithmic improvement. This physical language, in combination with uniqueness results related to unitarity, allow us to identify some potential pitfalls from kinetic energy fundamentally opposing the goal of optimization. This is connected to effects from wavefunction confinement, phase randomization, and shadow defects lurking in the objective far away from the ideal solution. As an example, we explore the surprising deficiency of many quantum methods in solving uncoupled spin problems and how this is both predictive of performance on some more complex systems while immediately suggesting simple resolutions. Further examination of canonical problems like the Hamming ramp or bush of implications show that entanglement can be strictly detrimental to performance results from the underlying mechanism of solution in approaches like QAOA. Kinetic energy and graph Laplacian perspectives provide new insights to common initialization and optimal solutions in QAOA as well as new methods for more effective layerwise training. Connections to classical methods of continuous extensions, homotopy methods, and iterated rounding suggest new directions for research in quantum optimization. Throughout, we unveil many pitfalls and mechanisms in quantum optimization using a physical perspective, which aim to spur the development of novel quantum optimization algorithms and refinements.

End-to-end learnable EEG channel selection for deep neural networks with Gumbel-softmax

Objective. To develop an efficient, embedded electroencephalogram (EEG) channel selection approach for deep neural networks, allowing us to match the channel selection to the target model, while avoiding the large computational burdens of wrapper approaches in conjunction with neural networks. Approach. We employ a concrete selector layer to jointly optimize the EEG channel selection and network parameters. This layer uses a Gumbel-softmax trick to build continuous relaxations of the discrete parameters involved in the selection process, allowing them be learned in an end-to-end manner with traditional backpropagation. As the selection layer was often observed to include the same channel twice in a certain selection, we propose a regularization function to mitigate this behavior. We validate this method on two different EEG tasks: motor execution and auditory attention decoding. For each task, we compare the performance of the Gumbel-softmax method with a baseline EEG channel selection approach tailored towards this specific task: mutual information and greedy forward selection with the utility metric respectively. Main results. Our experiments show that the proposed framework is generally applicable, while performing at least as well as (and often better than) these state-of-the-art, task-specific approaches. Significance. The proposed method offers an efficient, task- and model-independent approach to jointly learn the optimal EEG channels along with the neural network weights.

A decision space algorithm for multiobjective convex quadratic integer optimization

We present a branch-and-bound algorithm for minimizing multiple convex quadratic objective functions over integer variables. Our method looks for efficient points by fixing subsets of variables to integer values and by using lower bounds in the form of hyperplanes in the image space derived from the continuous relaxations of the restricted objective functions. We show that the algorithm stops after finitely many fixings of variables with detecting both the full efficient and the nondominated set of multiobjective strictly convex quadratic integer problems. A major advantage of the approach is that the expensive calculations are done in a preprocessing phase so that the nodes in the branch-and-bound tree can be enumerated fast. We show numerical experiments on biobjective instances and on instances with three and four objectives.

Mixed integer nonlinear optimization models for the Euclidean Steiner tree problem in $$\mathbb {R}^d$$

New mixed integer nonlinear optimization models for the Euclidean Steiner tree problem in d-space (with \(d\ge 3\)) will be presented in this work. All models feature a nonsmooth objective function but the continuous relaxations of their set of feasible solutions are convex. From these models, four convex mixed integer linear and nonlinear relaxations will be considered. Each relaxation has the same set of feasible solutions as the set of feasible solutions of the model from which it is derived. Finally, preliminary computational results highlighting the main features of the presented relaxations will be discussed.

Convergence of sum-up rounding schemes for cloaking problems governed by the Helmholtz equation

We consider the problem of designing a cloak for waves described by the Helmholtz equation from an integer programming point of view. The problem can be modeled as a PDE-constrained optimization problem with integer-valued control inputs that are distributed in the computational domain. A first-discretize-then-optimize approach results in a large-scale mixed-integer nonlinear program that is in general intractable because of the large number of integer variables that arise from the discretization of the domain. Instead, we propose an efficient algorithm that is able to approximate the local infima of the underlying nonconvex infinite-dimensional problem arbitrarily close without the need to solve the discretized finite-dimensional integer programs to optimality. We optimize only the continuous relaxations of the approximations for local minima and then apply the sum-up rounding methodology to obtain integer-valued controls. If the solutions of the discretized continuous relaxations converge to a local minimizer of the continuous relaxation, then the resulting discrete-valued control sequence converges weakly $$^*$$ in $$L^\infty$$ to the same local minimizer. These approximation properties follow under suitable refinements of the involved discretization grids. Our results use familiar concepts arising from the analytical properties of the underlying PDE and complement previous results, derived from a topology optimization point of view.

Sparse signal reconstruction for nonlinear models via piecewise rational optimization

We propose a method to reconstruct sparse signals degraded by a nonlinear distortion and acquired at a limited sampling rate. Our method formulates the reconstruction problem as a nonconvex minimization of the sum of a data fitting term and a penalization term. In contrast with most previous works which settle for approximated local solutions, we seek for a global solution to the obtained challenging nonconvex problem. Our global approach relies on the so-called Lasserre relaxation of polynomial optimization.We here specifically include in our approach the case of piecewise rational functions, which makes it possible to address a wide class of nonconvex exact and continuous relaxations of the ℓ0 penalization function. Additionally, we study the complexity of the optimization problem. It is shown how to use the structure of the problem to lighten the computational burden efficiently. Finally, numerical simulations illustrate the benefits of our method in terms of both global optimality and signal reconstruction.

Risk-Based Loan Pricing: Portfolio Optimization Approach with Marginal Risk Contribution

We consider a lender (bank) that determines the optimal loan price (interest rate) to offer to prospective borrowers under uncertain borrower response and default risk. A borrower may or may not accept the loan at the price offered, and both the principal loaned and the interest income become uncertain because of the risk of default. We present a risk-based loan pricing optimization framework that explicitly takes into account the marginal risk contribution, the portfolio risk, and a borrower’s acceptance probability. Marginal risk assesses the incremental risk contribution of a prospective loan to the bank’s overall portfolio risk by capturing the dependencies between the prospective loan and the existing portfolio and is evaluated with respect to the value-at-risk and conditional value-at-risk measures. We examine the properties and computational challenges of the formulations. We design a reformulation method based on the concavifiability concept to transform the nonlinear objective functions and to derive equivalent mixed-integer nonlinear reformulations with convex continuous relaxations. We also extend the approach to multiloan pricing problems, which feature explicit loan selection decisions in addition to pricing decisions. We derive formulations with multiple loans that take the form of mixed-integer nonlinear problems with nonconvex continuous relaxations and develop a computationally efficient algorithmic method. We provide numerical evidence demonstrating the value of the proposed framework, test the computational tractability, and discuss managerial implications. This paper was accepted by Chung Piaw Teo, optimization.