Separable Optimization Problems Research Articles

Multiscale dynamic graph signal analysis

We consider the problem of decomposing a time-varying graph signal into its constituent amplitude- and frequency-modulated components and inferring their dynamic functional connectivity structures in a data-driven manner, referred to as graph mode decomposition (GMD). The currently available method for GMD obtains static connectivity structures, limiting its applicability to real-life data. Further, the performance of the method relies critically on the accurate prior knowledge of the number of components to be extracted. We address those key limitations by presenting two approaches: (1) a two-stage method for obtaining the graph modes as well as corresponding connectivity structures by solving two separate optimization problems, and (2) a joint optimization formulation to achieve that task. The proposed optimization formulations include a prior to account for the time variation of graph edge weights. Moreover, we use a successive scheme to solve the problems which alleviates the limitation of having to specify the number of decomposed components a priori — improving the accuracy and data-driven capability while reducing the computational requirement of the methods. The performance of these methods is validated on a wide range of synthetic and real data sets. We also provide a public toolbox containing the MATLAB codes of the proposed methods.

Solving Sparse Separable Bilinear Programs Using Lifted Bilinear Cover Inequalities

Recently a class of second-order cone representable convex inequalities called lifted bilinear cover inequalities were introduced, which are valid for a set described by a separable bilinear constraint together with bounds on variables. In this paper, we study the computational potential of these inequalities for separable bilinear optimization problems. We first prove that the semidefinite programming relaxation provides no benefit over the McCormick relaxation for such problems. We then design a simple randomized separation heuristic for lifted bilinear cover inequalities. In our computational experiments, we separate many rounds of these inequalities starting from McCormick’s relaxation of instances where each constraint is a separable bilinear constraint set. We demonstrate that there is a significant improvement in the performance of a state-of-the-art global solver in terms of gap closed, when these inequalities are added at the root node compared with when they are not. History: Accepted by Andrea Lodi, Area Editor for Design & Analysis of Algorithms – Discrete. Funding: S. S. Dey gratefully acknowledges the support by the Office of Naval Research [Grant N000141912323].

Proximal Alternating Direction Method of Multipliers with Convex Combination Proximal Centers

Proximal alternating direction method of multipliers (PADMM) is a classical primal-dual splitting method for solving separable convex optimization problems with linear equality constraints, which have numerous applications in, e.g., signal and image processing, machine learning, and statistics. In this paper, we propose a new variant of PADMM, called PADMC, whose proximal centers are constructed by convex combinations of the iterates. PADMC is able to take advantage of problem structures and preserves the desirable properties of the classical PADMM. We establish iterate convergence as well as [Formula: see text] ergodic and [Formula: see text] nonergodic sublinear convergence rate results measured by function residual and feasibility violation, where [Formula: see text] denotes the iteration number. Moreover, we propose two fast variants of PADMC, one achieves faster [Formula: see text] ergodic convergence rate when one of the component functions is strongly convex, and the other ensures faster [Formula: see text] nonergodic convergence rate measured by constraint violation. Finally, preliminary numerical results on the LASSO and the elastic-net regularization problems are presented to demonstrate the performance of the proposed methods.

Analytical Placement with 3D Poisson’s Equation and ADMM-based Optimization for Large-scale 2.5D Heterogeneous FPGAs

As design complexity keeps increasing, the 2.5D field-programmable gate array (FPGA) with large logic capacity has become popular in modern circuit applications. A 2.5D FPGA consists of multiple dies connected through super long lines (SLLs) on an interposer. Each die contains heterogeneous logic blocks and ASIC-like clocking architectures to achieve better skew and timing. Existing works consider these problems separately and thus may lead to serious timing issues or routing failure. This article presents an analytical placement algorithm for the 2.5D FPGA to simultaneously minimize the number of inter-die SLL signals and intra-die clocking violations. Using a lifting dimension technique, we first formulate the 2.5D global placement problem as a three-dimensional continuous and differential minimization problem, where the SLL-aware block distribution is modeled by 3D Poisson’s equation and directly solved to obtain an analytical solution. Then, we further reformulate the minimization problem as a separable optimization problem with linear constraints. Based on the proximal alternating direction method of multipliers optimization method, we efficiently optimize the separable subproblems one by one in an alternating fashion. Finally, clock-aware legalization and detailed placement are applied to legalize and improve our placement results. Compared with the state-of-the-art works, experimental results show that our algorithm can resolve all clocking constraints and reduce the number of SLL crossing signals by 36.9% with similar wirelength in a comparable running time.

DIMIX: Diminishing Mixing for Sloppy Agents

We study nonconvex distributed optimization problems where a set of agents collaboratively solve a separable optimization problem that is distributed over a time-varying network. The existing methods to solve these problems rely on (at most) one-time-scale algorithms, where each agent performs a diminishing or constant step-size gradient descent at the average estimate of the agents in the network. However, if possible at all, exchanging exact information, which is required to evaluate these average estimates, potentially introduces a massive communication overhead. Therefore, a reasonable practical assumption to be made is that agents only receive a rough approximation of the neighboring agents’ information. To address this, we introduce and study a two-time-scale decentralized algorithm with a broad class of lossy information sharing methods (which includes noisy, quantized, and/or compressed information sharing) over time-varying networks. In our method, one time-scale suppresses the (imperfect) incoming information from the neighboring agents, and one time-scale operates on local cost functions’ gradients. We show that with a proper choices for the step-sizes’ parameters, the algorithm achieves a convergence rate of for nonconvex distributed optimization problems over time-varying networks for any .

Design of a optimization algorithm for binary classification

In the present work, the design of a system to classify data is carried out, using the Scrum methodology. The validation was carried out by expert judgment, having favorable results in terms of different criteria such as; integrity, ease of use, innovation, and scalability. Regarding the development of the functional elements of the system, it was obtained; he developed the architecture of the system, the database, and the prototypes, among other points considered. From the implementation of the system, the equation of a classifying plane in three-dimensional space will be obtained, as well as the number of internal iterations that the algorithm develops, the estimated execution time, and the graph of the plane. This system is based on a recently introduced symmetric cone proximal multiplier algorithm to solve separable optimization problems, this algorithm made an application for classification-related support vector machines.

Automatic differentiation rules for Tsoukalas–Mitsos convex relaxations in global process optimization

With increasing digitalization and vertical integration of chemical process systems, nonconvex optimization problems often emerge in chemical engineering applications, yet require specialized optimization techniques. Typical global optimization methods proceed by progressively refining bounds on the unknown optimal value, by strategically employing convex relaxations. This article constructs a general closed-form expression for the convex subdifferentials of recent “multivariate McCormick” convex relaxations of nontrivial composite functions, by solving a previous duality formulation in all cases using nonsmooth Karush–Kuhn–Tucker conditions. Based on this subdifferential expression, new automatic differentiation rules are developed to compute gradients and subgradients for multivariate McCormick relaxations, to ultimately generate useful bounds in global optimization. Unlike established differentiation techniques for these relaxations, our new rules are expressed in closed form, do not require solving separate dual optimization problems, are efficiently carried out, and are compatible with the reverse/adjoint mode of algorithmic differentiation. Our formulations become more straightforward when the relevant functions are either smooth or piecewise smooth.

Wind farm power production and fatigue load optimization based on dynamic partitioning and wake redirection of wind turbines

Due to the wake effect, there is a high degree of coupling between wind turbines in a wind farm. In order to optimize the power and load performance of wind farms, it is crucial to propose an optimization strategy to attenuate the wake interference between wind turbines by wake redirection. This study proposes a wake interference model that can consider wake deflection based on the yawed wake model. Accordingly, the interference relationship between turbines is represented as the topology of the graph. By solving for the connected components of the graph, the wind farm is divided into almost uncoupled partitions. In each partition, separate optimization problems with the objectives of power enhancement and load reduction are established. An improved Non-Dominated Sorting Genetic Algorithm with multiple crossover operators is used for solving the multi-objective optimization problem. Simulation results show that the proposed strategy can effectively lead to power enhancement and load reduction for the regular layout wind farm. For the curvilinear layout wind farm with an optimized layout considering wake effects, proposed strategy can further reduce wake disturbances to improve the performance of wind farms.

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

AbstractAlternating direction method of multipliers (ADMM) receives much attention in the field of optimization and computer science, etc. The generalized ADMM (G-ADMM) proposed by Eckstein and Bertsekas incorporates an acceleration factor and is more efficient than the original ADMM. However, G-ADMM is not applicable in some models where the objective function value (or its gradient) is computationally costly or even impossible to compute. In this paper, we consider the two-block separable convex optimization problem with linear constraints, where only noisy estimations of the gradient of the objective function are accessible. Under this setting, we propose a stochastic linearized generalized ADMM (called SLG-ADMM) where two subproblems are approximated by some linearization strategies. And in theory, we analyze the expected convergence rates and large deviation properties of SLG-ADMM. In particular, we show that the worst-case expected convergence rates of SLG-ADMM are $\mathcal{O}\left( {{N}^{-1/2}}\right)$ and $\mathcal{O}\left({\ln N} \cdot {N}^{-1}\right)$ for solving general convex and strongly convex problems, respectively, where N is the iteration number, similarly hereinafter, and with high probability, SLG-ADMM has $\mathcal{O}\left ( \ln N \cdot N^{-1/2} \right ) $ and $\mathcal{O}\left ( \left ( \ln N \right )^{2} \cdot N^{-1} \right ) $ constraint violation bounds and objective error bounds for general convex and strongly convex problems, respectively.

A Grouping Cooperative Differential Evolution Algorithm for Solving Partially Separable Complex Optimization Problems

A Grouping Cooperative Differential Evolution Algorithm for Solving Partially Separable Complex Optimization Problems

Synthesizing explainable counterfactual policies for algorithmic recourse with program synthesis

Being able to provide counterfactual interventions—sequences of actions we would have had to take for a desirable outcome to happen—is essential to explain how to change an unfavourable decision by a black-box machine learning model (e.g., being denied a loan request). Existing solutions have mainly focused on generating feasible interventions without providing explanations of their rationale. Moreover, they need to solve a separate optimization problem for each user. In this paper, we take a different approach and learn a program that outputs a sequence of explainable counterfactual actions given a user description and a causal graph. We leverage program synthesis techniques, reinforcement learning coupled with Monte Carlo Tree Search for efficient exploration, and rule learning to extract explanations for each recommended action. An experimental evaluation on synthetic and real-world datasets shows how our approach, FARE (eFficient counterfActual REcourse), generates effective interventions by making orders of magnitude fewer queries to the black-box classifier with respect to existing solutions, with the additional benefit of complementing them with interpretable explanations.

DBCC2: an improved difficulty-based cooperative co-evolution for many-modal optimization

Evolutionary multimodal optimization algorithms aim to provide multiple solutions simultaneously. Many studies have been conducted to design effective evolutionary algorithms for solving multimodal optimization problems. However, optimization problems with many global and acceptable local optima have not received much attention. This type of problem is undoubtedly challenging. In this study, we focus on problems with many optima, the so-called many-modal optimization problems, and this study is an extension of our previous conference work. First, a test suite including additively nonseparable many-modal optimization problems and partially additively separable many-modal optimization problems is designed. Second, an improved difficulty-based cooperative co-evolution algorithm (DBCC2) is proposed, which dynamically estimates the difficulties of subproblems and allocates the computational resources during the search. Experimental results show that DBCC2 has competitive performance.

SOLVING THE DAYS-OFF SCHEDULING PROBLEM USING QUADRATIC PROGRAMMING WITH CIRCULANT MATRIX

The purpose of this paper is the approach of a mathematical model dedicated to planning the consecutive days off of a company’s employees. Companies must find a flexible work schedule between employees, always considering the satisfaction of work tasks as well as guaranteeing consecutive days off. The analysis is based on solving a quadratic programming problem with binary variables. The proposed method uses the properties of the circulant symmetric matrix in the researched model, which allows the transformation of the considered problems into an equivalent separable non-convex optimization problem. A practical continuous convex relaxation approach is proposed. DC Algorithm is used to solve relaxed problems. A solved numerical example is presented.

A Multi-Step Inertial Proximal Peaceman-Rachford Splitting Method for Separable Convex Programming

In this paper, we propose a multi-step inertial proximal Peaceman-Rachford splitting method (abbreviated as MIP-PRSM) for solving the two-block separable convex optimization problems with linear constraints, which is a unified framework for such Peaceman-Rachford splitting methods (PRSM)-based improved algorithms with inertial step. Furthermore, we establish the global convergence of the MIP-PRSM under some assumptions. Finally, some numerical experimental results on the least squares semidefinite programming (LSSP), LASSO, the convex quadratic programming problem (CQPP), total variation (TV) based denoising and medical image reconstruction problems demonstrate the efficiency of the proposed method.

An inexact ADMM with proximal-indefinite term and larger stepsize

In this paper, an inexact Alternating Direction Method of Multipliers (ADMM) has been proposed for solving the two-block separable convex optimization problem subject to linear equality constraints. The first resulting subproblem is solved inexactly under relative error criterion, while another subproblem called regularization problem is solved inexactly by introducing an indefinite proximal term. Meanwhile, the dual variable is updated twice with relatively larger stepsizes since an indefinite proximal term is exploited. By reformulating the first-order optimality conditions of the involved subproblems as a monotone inclusion problem, the sublinear convergence rate of the proposed algorithm is established in the pointwise and ergodic sense. Numerical experiments on testing two sparse optimization problems from statistical learning and image restoration indicate that our inexact ADMM performs much better than several well-established algorithms.

An Iterative Implementation of Variable Projection for Separable Nonlinear Optimization Problems

The separable nonlinear least-squares (SNLLS) problems considered in this article frequently appear in a wide range of research fields, such as machine learning, computer vision, system identification, and signal processing. The variable projection algorithm proposed by Golub and Pereyra, which reduces the dimension of the parameters by projecting the linear parameters out of the problem, is quite valuable in solving SNLLS problems. Previous implementations of the variable projection algorithm are based on matrix factorization. In this article, we propose an iterative implementation of the variable projection algorithm. Compared with previous implementations based on matrix decomposition, the proposed method can effectively avoid suffering from large condition number of the matrix or even matrix decomposition failure when dealing with ill-posed SNLLS problems. Numerical experiments on real-world data and synthetic data show the efficiency and robustness of the proposed iterative variable projection algorithm.

Nuisance Parameter Estimation Algorithms for Separable Nonlinear Models

Many inverse problems in machine learning, system identification, and image processing include nuisance parameters, which are important for the recovering of other parameters. Separable nonlinear optimization problems fall into this category. The special separable structure in these problems has inspired several efficient optimization strategies. A well-known method is the variable projection (VP) that projects out a subset of the estimated parameters, resulting in a reduced problem that includes fewer parameters. The expectation maximization (EM) is another separated method that provides a powerful framework for the estimation of nuisance parameters. The relationships between EM and VP were ignored in previous studies, though they deal with a part of parameters in a similar way. In this article, we explore the internal relationships and differences between VP and EM. Unlike the algorithms that separate the parameters directly, the hierarchical identification algorithm decomposes a complex model into several linked submodels and identifies the corresponding parameters. Therefore, this article also studies the difference and connection between the hierarchical algorithm and the parameter-separated algorithms like VP and EM. In the numerical simulation part, Monte Carlo experiments are performed to further compare the performance of different algorithms. The results show that the VP algorithm usually converges faster than the other two algorithms and is more robust to the initial point of the parameters.

A Project Scheduling Game Equilibrium Problem Based on Dynamic Resource Supply

In a resource-constrained project scheduling problem, most studies ignore that resource supply is a separate optimization problem, which is not in line with the actual situation. In this study, the project scheduling problem and the resource supply problem are regarded as a dynamic game system, with interactive influences and constraints. This study proposes a Stackelberg dynamic game model based on the engineering supply chain perspective. In this model, the inherent conflicts and complex interactions between the Multi-mode Resource-Constrained Project Scheduling Problem (MRCPSP) and the Multi-Period Supply Chain Problem (MPSCP) are studied to determine the optimal equilibrium strategy. A two-level multi-objective programming method is used to solve the problem. The MRCPSP is the upper-level planning used to optimize project scheduling and activity mode selection to minimize project cost and duration; MPSCP is a lower-level planning method that seeks to make resource transportation decisions at a lower cost. A two-layer hybrid algorithm, consisting of Genetic Algorithm (GA) and Particle Swarm Optimization (PSO), is proposed to determine the optimal equilibrium strategy. Finally, the applicability and effectiveness of the proposed optimization method are evaluated through a case study of a large hydropower construction project, and management suggestions for related departments are provided.

Distributed Primal–Dual Splitting Algorithm for Multiblock Separable Optimization Problems

This article considers the distributed structured optimization problem of collaboratively minimizing the global objective function composed of the sum of local cost functions. Each local objective function involves a Lipschitz-differentiable convex function, a nonsmooth convex function, and a linear composite nonsmooth convex function. For such problems, we derive the synchronous distributed primal–dual splitting (S-DPDS) algorithm with uncoordinated stepsizes. Meanwhile, we develop the asynchronous version of the algorithm in light of the randomized block-coordinate method (A-DPDS). Further, the convergence results show the relaxed range and concise form of the acceptable parameters, which indicates that the algorithms are conducive to the selection of parameters in practical applications. Finally, we demonstrate the efficiency of S-DPDS and A-DPDS algorithms by numerical experiments.

Stable Bounds on the Duality Gap of Separable Nonconvex Optimization Problems

The Shapley-Folkman theorem shows that Minkowski averages of uniformly bounded sets tend to be convex when the number of terms in the sum becomes much larger than the ambient dimension. In optimization, Aubin and Ekeland show that this produces an a priori bound on the duality gap of separable nonconvex optimization problems involving finite sums. This bound is highly conservative and depends on unstable quantities; we relax it in several directions to show that nonconvexity can have a much milder impact on finite sum minimization problems, such as empirical risk minimization and multitask classification. As a byproduct, we show a new version of Maurey’s classical approximate Carathéodory lemma where we sample a significant fraction of the coefficients, without replacement, as well as a result on sampling constraints using an approximate Helly theorem, both of independent interest. Funding: A. d’Aspremont acknowledges support from the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program [Grant ANR-19-P3IA-0001] (PRAIRIE 3IA Institute), the Machine Learning and Optimisation Joint Research Initiative with the Fonds AXA pour la Recherche and Kamet Ventures, as well as a Google focused award.