Convex-concave Optimization Research Articles

Dynamic Environment Responsive Online Meta-Learning with Fairness Awareness

The fairness-aware online learning framework has emerged as a potent tool within the context of continuous lifelong learning. In this scenario, the learner’s objective is to progressively acquire new tasks as they arrive over time, while also guaranteeing statistical parity among various protected sub-populations, such as race and gender when it comes to the newly introduced tasks. A significant limitation of current approaches lies in their heavy reliance on the i.i.d (independent and identically distributed) assumption concerning data, leading to a static regret analysis of the framework. Nevertheless, it’s crucial to note that achieving low static regret does not necessarily translate to strong performance in dynamic environments characterized by tasks sampled from diverse distributions. In this article, to tackle the fairness-aware online learning challenge in evolving settings, we introduce a unique regret measure, FairSAR, by incorporating long-term fairness constraints into a strongly adapted loss regret framework. Moreover, to determine an optimal model parameter at each time step, we introduce an innovative adaptive fairness-aware online meta-learning algorithm, referred to as FairSAOML. This algorithm possesses the ability to adjust to dynamic environments by effectively managing bias control and model accuracy. The problem is framed as a bi-level convex-concave optimization, considering both the model’s primal and dual parameters, which pertain to its accuracy and fairness attributes, respectively. Theoretical analysis yields sub-linear upper bounds for both loss regret and the cumulative violation of fairness constraints. Our experimental evaluation of various real-world datasets in dynamic environments demonstrates that our proposed FairSAOML algorithm consistently outperforms alternative approaches rooted in the most advanced prior online learning methods.

Local AdaGrad-type algorithm for stochastic convex-concave optimization

Large scale convex-concave minimax problems arise in numerous applications, including game theory, robust training, and training of generative adversarial networks. Despite their wide applicability, solving such problems efficiently and effectively is challenging in the presence of large amounts of data using existing stochastic minimax methods. We study a class of stochastic minimax methods and develop a communication-efficient distributed stochastic extragradient algorithm, LocalAdaSEG, with an adaptive learning rate suitable for solving convex-concave minimax problems in the Parameter-Server model. LocalAdaSEG has three main features: (1) a periodic communication strategy that reduces the communication cost between workers and the server; (2) an adaptive learning rate that is computed locally and allows for tuning-free implementation; and (3) theoretically, a nearly linear speed-up with respect to the dominant variance term, arising from the estimation of the stochastic gradient, is proven in both the smooth and nonsmooth convex-concave settings. LocalAdaSEG is used to solve a stochastic bilinear game, and train a generative adversarial network. We compare LocalAdaSEG against several existing optimizers for minimax problems and demonstrate its efficacy through several experiments in both homogeneous and heterogeneous settings.

Utilizing additive manufacturing and mass customization under capacity constraints

Additive manufacturing (AM), originally used for prototyping, is increasingly adopted for custom final part production across different industries. However, printing speed and production volume are two barriers for the adoption of AM for product customization at large scale. Nevertheless, manufacturers could aim to combine the benefits of AM for product customization with traditional mass customization (MC) technologies over the product life cycle (PLC). This approach is showcased in our paper as a manufacturing opportunity and is addressed via a non convex-concave optimization model that considers a monopolist manufacturer producing horizontally differentiated products at scale. To satisfy individual customer preferences under capacity considerations, the firm jointly decides on the inventory, production quantity, product variety, optimal technology-switching times (between AM and MC) and pricing strategy. Our approach can be implemented by decision-makers to leverage customer-centricity and benefit from this novel hybrid manufacturing practice. By deriving a closed-form solution for the production quantity based on an adaptive inventory policy, the resulting optimization problem is solved using the Sample Average Approximation framework grounded by analytical results. Our results demonstrate that the new usage of AM with MC can benefit a manufacturer for customer-centric driven strategies. Significant profit improvements can be achieved with an AM–MC–AM technology-switching scenario under certain capacity conditions and with an increasing-decreasing pricing strategy. Our results also indicate that the benefits of pricing flexibility are highest when capacity is unlimited or when the firm does not hold inventory. Under capacity constraints, a simple decreasing pricing policy combined with inventory performs very well.

Constrained Optimal PID Controller Design: Convex-Concave Optimization Approach

Constrained Optimal PID Controller Design: Convex-Concave Optimization Approach

Partial Label Learning with Self-Guided Retraining

Partial label learning deals with the problem where each training instance is assigned a set of candidate labels, only one of which is correct. This paper provides the first attempt to leverage the idea of self-training for dealing with partially labeled examples. Specifically, we propose a unified formulation with proper constraints to train the desired model and perform pseudo-labeling jointly. For pseudo-labeling, unlike traditional self-training that manually differentiates the ground-truth label with enough high confidence, we introduce the maximum infinity norm regularization on the modeling outputs to automatically achieve this consideratum, which results in a convex-concave optimization problem. We show that optimizing this convex-concave problem is equivalent to solving a set of quadratic programming (QP) problems. By proposing an upper-bound surrogate objective function, we turn to solving only one QP problem for improving the optimization efficiency. Extensive experiments on synthesized and real-world datasets demonstrate that the proposed approach significantly outperforms the state-of-the-art partial label learning approaches.

Precise Error Analysis of Regularized <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math> </inline-formula>-Estimators in High Dimensions

A popular approach for estimating an unknown signal $ \mathbf {x}_{0}\in \mathbb {R} ^{n}$ from noisy, linear measurements $ \mathbf {y}= \mathbf {A} \mathbf {x} _{0}+ \mathbf {z}\in \mathbb {R}^{m}$ is via solving a so called regularized $M$ -estimator: $\hat{\mathbf {x}} :=\arg \min _ \mathbf {x} \mathcal {L} (\mathbf {y}- \mathbf {A} \mathbf {x})+\lambda f(\mathbf {x})$ . Here, $ \mathcal {L}$ is a convex loss function, $f$ is a convex (typically, non-smooth) regularizer, and $\lambda > 0$ is a regularizer parameter. We analyze the squared error performance $\|\hat{\mathbf {x}} - \mathbf {x}_{0}\|_{2}^{2}$ of such estimators in the high-dimensional proportional regime where $m,n\rightarrow \infty $ and $m/n\rightarrow \delta $ . The design matrix $ \mathbf {A}$ is assumed to have entries iid Gaussian; only minimal and rather mild regularity conditions are imposed on the loss function, the regularizer, and on the noise and signal distributions. We show that the squared error converges in probability to a nontrivial limit that is given as the solution to a minimax convex-concave optimization problem on four scalar optimization variables. We identify a new summary parameter, termed the expected Moreau envelope to play a central role in the error characterization. The precise nature of the results permits an accurate performance comparison between different instances of regularized $M$ -estimators and allows to optimally tune the involved parameters (such as the regularizer parameter and the number of measurements). The key ingredient of our proof is the convex Gaussian min-max theorem which is a tight and strengthened version of a classical Gaussian comparison inequality that was proved by Gordon in 1988.

A data-driven approach to robust control of multivariable systems by convex optimization

The frequency-domain data of a multivariable system in different operating points is used to design a robust controller with respect to the measurement noise and multimodel uncertainty. The controller is fully parameterized in terms of matrix polynomial functions and can be formulated as a centralized, decentralized or distributed controller. All standard performance specifications like H2, H∞ and loop shaping are considered in a unified framework for continuous- and discrete-time systems. The control problem is formulated as a convex–concave optimization problem and then convexified by linearization of the concave part around an initial controller. The performance criterion converges monotonically to a local optimum or a saddle point in an iterative algorithm. The effectiveness of the method is compared with fixed-structure controller design methods based on non-smooth optimization via multiple simulation examples.

Robust PID Design Based on QFT and Convex–Concave Optimization

This paper presents an automatic loop-shaping method for designing proportional integral derivative controllers. Criteria for load disturbance attenuation, measurement noise injection, set-point response and robustness to plant uncertainty are given. One criterion is chosen to be optimized with the remaining ones as constraints. Two cases are considered: M-constrained integral gain optimization and minimization of the cost of feedback according to quantitative feedback theory. Optimization is performed using a convex–concave procedure (CCP). The method that relies on solving a sequence of convex optimization problems converges to a local minimum or a saddle point. The proposed method is illustrated by examples.

Saddle point mirror descent algorithm for the robust PageRank problem

In order to solve robust PageRank problem a saddle-point Mirror Descent algorithm for solving convex-concave optimization problems is enhanced and studied. The algorithm is based on two proxy functions, which use specificities of value sets to be optimized on (min-max search). In robust PageRank case the ones are entropy-like function and square of Euclidean norm. The saddle-point Mirror Descent algorithm application to robust PageRank leads to concrete complexity results, which are being discussed alongside with illustrative numerical example.

An algorithm for constrained one-step inversion of spectral CT data

We develop a primal-dual algorithm that allows for one-step inversion of spectral CT transmission photon counts data to a basis map decomposition. The algorithm allows for image constraints to be enforced on the basis maps during the inversion. The derivation of the algorithm makes use of a local upper bounding quadratic approximation to generate descent steps for non-convex spectral CT data discrepancy terms, combined with a new convex-concave optimization algorithm. Convergence of the algorithm is demonstrated on simulated spectral CT data. Simulations with noise and anthropomorphic phantoms show examples of how to employ the constrained one-step algorithm for spectral CT data.

PID Controller Auto Tuning using ASBO Technique

Despite the popularity and usability of proportional- integral-derivative (PID) controllers in industrial applications, still its auto tuning is a challenge for researchers in terms of applied algorithmic efficiency and optimal definition of performance index. With a belief, compare to other species society, at present human society is more properly organized and developed; a new optimization method based on human social behavior called adaptive social behavior optimization (ASBO) has applied in this paper to auto tune the PID controller parameters in regards to achieving the global solution. A robust fitness function as performance index has also designed to get better exploration for optimal tuning in terms of various performance parameters. Experiments have given, with a number of systems having different types of characteristics and complexity like quadrotor, automatic voltage regulator (AVR) and DC motor systems etc. To understand the relative benefits of the proposed method, performance comparison with the number of other frequently applied algorithms in literatures like Genetic algorithm (GA) and its variant called self organize genetic algorithm (SOGA), Differential Evolution (DE), an extension of probabilistic distribution based Chaotic estimation of distribution algorithms (CEDA),Chaotic optimization and Convex-Concave optimization have presented with various practically applied fitness criteria's in practice. Proposed method of auto tuning has shown the generalized applicability for PID controller design with different types of systems in optimum manner.

Optimization of Interference Alignment Beamforming Vectors

The K user interference alignment scheme proposed by Cadambe and Jafar (CJ) achieves K/2 degrees of freedom for networks with frequency selective channels. In this paper, we provide two new algorithms that optimize the precoding subspaces which maximize the data rate performance of the CJ scheme while maintaining the achievable degrees of freedom. One design is obtained as a global solution of a constrained convex (concave) optimization problem that maximizes the sum rate. The other design provides a low complexity closed-form solution to a constrained maximization problem with a suboptimal sum rate objective function. We also show that both designs can be combined with Shen, Host-Madsen and Vidal orthonormalization that achieves further gains in sum rate by optimizing the precoding vectors generated by the proposed algorithms.

Discriminative Semi-Supervised Feature Selection Via Manifold Regularization

Feature selection has attracted a huge amount of interest in both research and application communities of data mining. We consider the problem of semi-supervised feature selection, where we are given a small amount of labeled examples and a large amount of unlabeled examples. Since a small number of labeled samples are usually insufficient for identifying the relevant features, the critical problem arising from semi-supervised feature selection is how to take advantage of the information underneath the unlabeled data. To address this problem, we propose a novel discriminative semi-supervised feature selection method based on the idea of manifold regularization. The proposed approach selects features through maximizing the classification margin between different classes and simultaneously exploiting the geometry of the probability distribution that generates both labeled and unlabeled data. In comparison with previous semi-supervised feature selection algorithms, our proposed semi-supervised feature selection method is an embedded feature selection method and is able to find more discriminative features. We formulate the proposed feature selection method into a convex-concave optimization problem, where the saddle point corresponds to the optimal solution. To find the optimal solution, the level method, a fairly recent optimization method, is employed. We also present a theoretic proof of the convergence rate for the application of the level method to our problem. Empirical evaluation on several benchmark data sets demonstrates the effectiveness of the proposed semi-supervised feature selection method.