Convex Optimization Problem Research Articles

An asynchronous proximal bundle method

AbstractWe develop a fully asynchronous proximal bundle method for solving non-smooth, convex optimization problems. The algorithm can be used as a drop-in replacement for classic bundle methods, i.e., the function must be given by a first-order oracle for computing function values and subgradients. The algorithm allows for an arbitrary number of master problem processes computing new candidate points and oracle processes evaluating functions at those candidate points. These processes share information by communication with a single supervisor process that resembles the main loop of a classic bundle method. All processes run in parallel and no explicit synchronization step is required. Instead, the asynchronous and possibly outdated results of the oracle computations can be seen as an inexact function oracle. Hence, we show the convergence of our method under weak assumptions very similar to inexact and incremental bundle methods. In particular, we show how the algorithm learns important structural properties of the functions to control the inaccuracy induced by the asynchronicity automatically such that overall convergence can be guaranteed.

Variance-Reduced Accelerated First-Order Methods: Central Limit Theorems and Confidence Statements

In this paper, we consider a strongly convex stochastic optimization problem and propose three classes of variable sample-size stochastic first-order methods: (i) the standard stochastic gradient descent method, (ii) its accelerated variant, and (iii) the stochastic heavy-ball method. In each scheme, the exact gradients are approximated by averaging across an increasing batch size of sampled gradients. We prove that when the sample size increases at a geometric rate, the generated estimates converge in mean to the optimal solution at an analogous geometric rate for schemes (i)–(iii). Based on this result, we provide central limit statements, whereby it is shown that the rescaled estimation errors converge in distribution to a normal distribution with the associated covariance matrix dependent on the Hessian matrix, the covariance of the gradient noise, and the step length. If the sample size increases at a polynomial rate, we show that the estimation errors decay at a corresponding polynomial rate and establish the associated central limit theorems (CLTs). Under certain conditions, we discuss how both the algorithms and the associated limit theorems may be extended to constrained and nonsmooth regimes. Finally, we provide an avenue to construct confidence regions for the optimal solution based on the established CLTs and test the theoretical findings on a stochastic parameter estimation problem. Funding: This work was supported by the Office of Naval Research [Grant N00014-22-1-2589] and Basic Energy Sciences [Grant DE-SC0023303]. The work of J. Lei was partially supported by the National Natural Science Foundation of China [Grants 72271187 and 62088101]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2021.0068 .

Dynamic event‐triggered fault estimation and accommodation for networked systems based on intermediate variable

AbstractIn this paper, the problem of robust fault estimation and fault‐tolerant control is considered for networked systems with external disturbances. By introducing an intermediate variable containing the system state information and fault information, the observer to accurately estimate the fault and system state is proposed. The gain matrices of fault estimation observer, fault‐tolerant controller and dynamic event‐triggered mechanism(ETM) are calculated simultaneously by solving a convex optimization problem. Compared with the existing results, the proposed fault estimation strategy does not match observer matching conditions. Furthermore, the introduction of dynamic ETM can effectively reduce the burden of the network with limited bandwidth resources. Finally, two simulation examples show the effectiveness of the proposed method.

Switching Anti-Windup Synthesis for Linear Systems With Asymmetric Actuator Saturation.

This article proposes a switching anti-windup strategy for linear, time-invariant (LTI) systems subject to asymmetric actuator saturation and L2 -disturbances, the core idea behind which is to make full use of the available range of control input space by switching among multiple anti-windup gains. The asymmetrically saturated LTI system is converted to a switched system with symmetrically saturated subsystems, and a dwell time switching rule is presented to govern the switching between different antiwindup gains. Based on multiple Lyapunov functions, we derive sufficient conditions for guaranteeing the regional stability and weighted L2 performance of the closed-loop system. The switching anti-windup synthesis that designs a separate anti-windup gain for each subsystem is cast as a convex optimization problem. In comparison with the design of a single anti-windup gain, our method can induce less conservative results since the asymmetric character of the saturation constraint is fully utilized in the switching anti-windup design. Two numerical examples, and an application to aeroengine control (the experiments are conducted on a semiphysical test bench), demonstrate the superiority and practicality of the proposed scheme.

Asynchronous Parallel Large-Scale Gaussian Process Regression.

Gaussian process regression (GPR) is an important nonparametric learning method in machine learning research with many real-world applications. It is well known that training large-scale GPR is a challenging task due to the required heavy computational cost and large volume memory. To address this challenging problem, in this article, we propose an asynchronous doubly stochastic gradient algorithm to handle the large-scale training of GPR. We formulate the GPR to a convex optimization problem, i.e., kernel ridge regression. After that, in order to efficiently solve this convex kernel problem, we first use the random feature mapping method to approximate the kernel model and then utilize two unbiased stochastic approximations, i.e., stochastic variance reduced gradient and stochastic coordinate descent, to update the solution asynchronously and in parallel. In this way, our algorithm scales well in both sample size and dimensionality, and speeds up the training computation. More importantly, we prove that our algorithm has a global linear convergence rate. Our experimental results on eight large-scale benchmark datasets with both regression and classification tasks show that the proposed algorithm outperforms the existing state-of-the-art GPR methods.

Dynamical System Approach for Time-Varying Constrained Convex Optimization Problems

Dynamical System Approach for Time-Varying Constrained Convex Optimization Problems

RECURRENT NEURAL NETWORK LEARNING ALGORITHM-BASED EVENT-TRIGGERED OBSERVER OF THE PERMANENT MAGNET SYNCHRONOUS MOTOR

In this paper, we propose a novel method to estimate the electrical angular velocity, the electrical angle, and the currents of the permanent magnet synchronous motor. A recurrent neural network learning algorithm is first developed to estimate the states of the permanent magnet synchronous motor. Then, an event-triggered state observer is designed for the recurrent neural network. This state observer robustly estimates state variables of the permanent magnet synchronous motor. A sufficient condition in terms of a convex optimization problem for the existence of the event-triggered state observer is established. In contrast with the abundance of state estimation methods based on time-triggered state observers where the measurements are always continuously available, the ones in this paper are updated when an event-triggered condition holds. Therefore, it lessens the stress on communication resources while can still maintain an estimation performance. Simulation results are provided to demonstrate the merit of the proposed method.

Distributed online constrained convex optimization with event-triggered communication

This paper focuses on the distributed online convex optimization problem with time-varying inequality constraints over a network of agents, where each agent collaborates with its neighboring agents to minimize the cumulative network-wide loss over time. To reduce communication overhead between the agents, we propose a distributed event-triggered online primal–dual algorithm over a time-varying directed graph. With several classes of appropriately chose decreasing parameter sequences and non-increasing event-triggered threshold sequences, we establish dynamic network regret and network cumulative constraint violation bounds. Finally, a numerical simulation example is provided to verify the theoretical results.

Accelerated Primal-Dual Projection Neurodynamic Approach With Time Scaling for Linear and Set Constrained Convex Optimization Problems

Accelerated Primal-Dual Projection Neurodynamic Approach With Time Scaling for Linear and Set Constrained Convex Optimization Problems

Maximum likelihood weight estimation for partial domain adaptation

Partial Domain Adaptation (PDA) aims to generalize a classification model from a labeled source domain to an unlabeled target domain, where the source label space contains the target label space. There are two main challenges in PDA that weaken the model's classification performance in the target domain: (i) the joint distribution of the source domain is related but different from that of the target domain, and (ii) the source outlier data, whose labels do not belong to the target label space, have a negative impact on learning the target classification model. To tackle these challenges, we propose a Maximum Likelihood Weight Estimation (MLWE) approach to estimate a weight function for the source domain. The weight function matches the joint source distribution of the relevant part to the joint target distribution, and reduces the negative impact of the source outlier data. To be specific, our approach estimates the weight function by maximizing a likelihood function, and the estimation leads to a nice convex optimization problem that has a global optimal solution. In the experiments, our approach demonstrates superior performance on popular benchmark datasets. Intro video and PyTorch code are available at https://github.com/sentaochen/Maximum-Likelihood-Weight-Estimation.

Multivariate, heteroscedastic empirical Bayes via nonparametric maximum likelihood

Abstract Multivariate, heteroscedastic errors complicate statistical inference in many large-scale denoizing problems. Empirical Bayes is attractive in such settings, but standard parametric approaches rest on assumptions about the form of the prior distribution which can be hard to justify and which introduce unnecessary tuning parameters. We extend the nonparametric maximum-likelihood estimator (NPMLE) for Gaussian location mixture densities to allow for multivariate, heteroscedastic errors. NPMLEs estimate an arbitrary prior by solving an infinite-dimensional, convex optimization problem; we show that this convex optimization problem can be tractably approximated by a finite-dimensional version. The empirical Bayes posterior means based on an NPMLE have low regret, meaning they closely target the oracle posterior means one would compute with the true prior in hand. We prove an oracle inequality implying that the empirical Bayes estimator performs at nearly the optimal level (up to logarithmic factors) for denoizing without prior knowledge. We provide finite-sample bounds on the average Hellinger accuracy of an NPMLE for estimating the marginal densities of the observations. We also demonstrate the adaptive and nearly optimal properties of NPMLEs for deconvolution. We apply our method to two denoizing problems in astronomy and to two hierarchical linear modelling problems in social science and biology.

An improved convex optimization-based guidance for fuel-optimal powered landing

Convex optimization has great potential for onboard trajectory generation which can eliminate large state deviations and improve landing precision. In practical applications, the convex optimization problem is often infeasible under disturbances. In this paper, we analyze the reason for the infeasibility of the fuel-optimal trajectory optimization problem. The analysis result shows that the thrust saturation degrade the anti-disturbance performance and then lead to infeasibility. Based on this analysis, we improved the anti-disturbance performance of convex optimization-based guidance from two aspects: (1) a maximum-thrust regulation strategy is embedded into the original fuel-optimal problem to avoid thrust saturation; (2) a trajectory tracking method with a disturbance compensator is performed to attenuate disturbance effects and to make the initial state of each optimization cycle close to the existing feasible trajectory. Numerical simulations demonstrate that the proposed method can improve the infeasibility and can realize the high-precision landing under disturbances.

Statistically consistent inverse optimal control for discrete-time indefinite linear–quadratic systems

The Inverse Optimal Control (IOC) problem is a structured system identification problem that aims to identify the underlying objective function based on observed optimal trajectories. This provides a data-driven way to model experts’ behavior. In this paper, we consider the case of discrete-time finite-horizon linear–quadratic problems where: the quadratic cost term in the objective is not necessarily positive semi-definite; the planning horizon is a random variable; we have both process noise and observation noise; the dynamics can have a drift term; and where we can have a linear cost term in the objective. In this setting, we first formulate the necessary and sufficient conditions for when the forward optimal control problem is solvable. Next, we show that the corresponding IOC problem is identifiable. Using the conditions for existence of an optimum of the forward problem, we then formulate an estimator for the parameters in the objective function of the forward problem as the globally optimal solution to a convex optimization problem, and prove that the estimator is statistical consistent. Finally, the performance of the algorithm is demonstrated on two numerical examples.

A convex level-set method with multiplicative-additive model for image segmentation

The existing active contour models (ACMs) based on bias field (BF) correction mostly rely on a single BF assumption and lack in-depth discussion on the convexity of the energy functional, often leading to the problem of local minima. To address this issue, this paper introduces a dual BF and proposes a convex level-set (LS) method based on multiplicative-additive (MA) model to achieve global minima. Firstly, a MA model is adopted as the fidelity term, and a kernel function is introduced to adjust the size of the intensity inhomogeneous neighborhood, enhancing the adaptability to intensity inhomogeneity. Then, the convex LS function is embedded in the variational framework to ensure convexity of each variable in the energy functional. This transformation turns the segmentation problem into a convex optimization problem. By introducing the total variation regularization term to smooth the LS function, the model's resistance to noise is effectively enhanced. Finally, by minimizing the proposed energy functional, image segmentation and BF correction are successfully achieved. Experimental results validate the global minima property of our model, while also demonstrating good flexibility in the initial contour. The proposed model achieves superior segmentation results compared to other classical ACMs on various types of images.

Probability‐guaranteed encoding–decoding‐based state estimation for delayed memristive neutral networks with event‐triggered mechanism

SummaryThis article handles the probability‐guaranteed state estimation problem for a class of nonlinear memristive neural networks (MNNs) by using an event‐triggered mechanism. Both time‐varying delays and incomplete measurements are considered in the MNNs dynamics. To mitigate the impact of limited communication bandwidth, a communication protocol is proposed that incorporates an encoding–decoding technique in addition to an event‐triggered scheme. The aim is to devise a state estimator that can estimate the states of MNNs, ensuring that the state estimation error falls within the required ellipsoidal area with a desired chance. We obtain sufficient conditions for the feasibility of the addressed problem, where the requested gains can be found iteratively by solving certain convex optimization problems. On the basis of the proposed framework, some issues are further presented to determine locally optimal estimator parameters according to different specifications. Finally, we utilize an illustrative numerical example to show the validity of our provided theoretical results.

Single-Projection Procedure for Infinite Dimensional Convex Optimization Problems

Single-Projection Procedure for Infinite Dimensional Convex Optimization Problems

Using Convex Optimization to Efficiently Apportion Tracer and Pollutant Sources From Point Concentration Observations

AbstractRivers transport elements, minerals, chemicals, and pollutants produced in their upstream basins. A sample from a river is a mixture of all of its upstream sources, making it challenging to pinpoint the contribution from each individual source. Here, we show how a nested sample design and convex optimization can be used to efficiently unmix downstream samples of a well‐mixed, conservative tracer in a steady state system into the contributions of their upstream sources. Our approach is significantly faster than previous methods. We represent the river's sub‐catchments, defined by sampling sites, using a directed acyclic graph. This graph is used to build a convex optimization problem which, thanks to its convexity, can be quickly solved to global optimality—in under a second on desktop hardware for data sets of ∼100 samples or fewer. Uncertainties in the upstream predictions can be generated using Monte Carlo resampling. We provide an open‐source implementation of this approach in Python. The inputs required are straightforward: a table containing sample locations and observed tracer concentrations, along with a D8 flow‐direction raster map. As a case study, we use this method to map the elemental geochemistry of sediment sources for rivers draining the Cairngorms mountains, UK. This method could be extended to non‐conservative and non‐steady state tracers. We also show, theoretically, how multiple tracers could be simultaneously inverted to recover upstream run‐off or erosion rates as well as source concentrations. Overall, this approach can provide valuable insights to researchers in various fields, including water quality, geochemical exploration, geochemistry, hydrology, and wastewater epidemiology.

A new computational framework for log-concave density estimation

In statistics, log-concave density estimation is a central problem within the field of nonparametric inference under shape constraints. Despite great progress in recent years on the statistical theory of the canonical estimator, namely the log-concave maximum likelihood estimator, adoption of this method has been hampered by the complexities of the non-smooth convex optimization problem that underpins its computation. We provide enhanced understanding of the structural properties of this optimization problem, which motivates the proposal of new algorithms, based on both randomized and Nesterov smoothing, combined with an appropriate integral discretization of increasing accuracy. We prove that these methods enjoy, both with high probability and in expectation, a convergence rate of order 1/T up to logarithmic factors on the objective function scale, where T denotes the number of iterations. The benefits of our new computational framework are demonstrated on both synthetic and real data, and our implementation is available in a github repository LogConcComp (Log-Concave Computation).

Iterative reconstruction for limited-angle CT using implicit neural representation

Objective. Limited-angle computed tomography (CT) presents a challenge due to its ill-posed nature. In such scenarios, analytical reconstruction methods often exhibit severe artifacts. To tackle this inverse problem, several supervised deep learning-based approaches have been proposed. However, they are constrained by limitations such as generalization issue and the difficulty of acquiring a large amount of paired CT images. Approach. In this work, we propose an iterative neural reconstruction framework designed for limited-angle CT. By leveraging a coordinate-based neural representation, we formulate tomographic reconstruction as a convex optimization problem involving a deep neural network. We then employ differentiable projection layer to optimize this network by minimizing the discrepancy between the predicted and measured projection data. In addition, we introduce a prior-based weight initialization method to ensure the network starts optimization with an informed initial guess. This strategic initialization significantly improves the quality of iterative reconstruction by stabilizing the divergent behavior in ill-posed neural fields. Our method operates in a self-supervised manner, thereby eliminating the need for extensive data. Main results. The proposed method outperforms other iterative and learning-based methods. Experimental results on XCAT and Mayo Clinic datasets demonstrate the effectiveness of our approach in restoring anatomical features as well as structures. This finding was substantiated by visual inspections and quantitative evaluations using NRMSE, PSNR, and SSIM. Moreover, we conduct a comprehensive investigation into the divergent behavior of iterative neural reconstruction, thus revealing its suboptimal convergence when starting from scratch. In contrast, our method consistently produced accurate images by incorporating an initial estimate as informed initialization. Significance. This work showcases the feasibility to reconstruct high-fidelity CT images from limited-angle x-ray projections. The proposed methodology introduces a novel data-free approach to enhance medical imaging, holding promise across various clinical applications.

On the Bregman-proximal iterative algorithm for the monotone inclusion problem in Banach spaces

In this paper, we focus on the solution of a class of monotone inclusion problems in reflexive Banach spaces. To reflect the geometry of the space and the operator, a more general proximal point iterative algorithm with Bregman divergence is proposed and some strong convergence results for the proposed scheme under standard assumptions are obtained. Meanwhile, the convex optimization problems and the critical point problems are studied in the applications, and the recovery of the sparse signal is simulated in the numerical experiments.