Augmented Lagrangian Function Research Articles

A noise statistical distribution analysis-based two-step filtering mechanism for optical coherence tomography image despeckling

Optical coherence tomography (OCT) inevitably suffers from speckle noises that originate from the coherent multiple-scattered photons. Such speckle noises, following different distribution patterns, can hide tissue microstructures and degrade the disease diagnosis accuracy. So far, various schemes have been proposed for despeckling in OCT images, yet few have evaluated the impacts of different noise patterns on despeckling effects. In this study, we evaluate the influences of statistical noise distributions on OCT despeckling and propose a noise distribution analysis-based despeckling method for OCT images. Specifically, we establish a noise model by dividing speckle noises into multiplicative and additive ones first, and then propose a two-step filtering mechanism, namely, augmented Lagrange function minimization and Rayleigh alpha-trimmed filtering (AR) method, to suppress such noises separately while maintain tissue microstructures. Simulations with both synthetic and practical OCT images are conducted to verify the effectiveness of the proposed AR method. Results show that the influence of multiplicative noises on OCT images are more significant, and the AR method is capable of suppressing both multiplicative and additive noises effectively, e.g. it improves the peak signal-to-noise ratio and structural similarity index measurements by 83.22 and 812.88 for typical retinal images, respectively, while retaining the important image structural details with less computational time.

DiNNO: Distributed Neural Network Optimization for Multi-Robot Collaborative Learning

We present a distributed algorithm that enables a group of robots to collaboratively optimize the parameters of a deep neural network model while communicating over a mesh network. Each robot only has access to its own data and maintains its own version of the neural network, but eventually learns a model that is as good as if it had been trained on all the data centrally. No robot sends raw data over the wireless network, preserving data privacy and ensuring efficient use of wireless bandwidth. At each iteration, each robot approximately optimizes an augmented Lagrangian function, then communicates the resulting weights to its neighbors, updates dual variables, and repeats. Eventually, all robots' local network weights reach a consensus. For convex objective functions, we prove this consensus is a global optimum. We compare our algorithm to two existing distributed deep neural network training algorithms in (i) an MNIST image classification task, (ii) a multi-robot implicit mapping task, and (iii) a multi-robot reinforcement learning task. In all of our experiments our method out performed baselines, and was able to achieve validation loss equivalent to centrally trained models. See \href{https://msl.stanford.edu/projects/dist_nn_train}{https://msl.stanford.edu/projects/dist\_nn\_train} for videos and a link to our GitHub repository.

An ADMM Based Parallel Approach for Fund of Fund Construction

In this paper, we propose a parallel algorithm for a fund of fund (FOF) optimization model. Based on the structure of objective function, we create an augmented Lagrangian function and separate the quadratic term from the nonlinear term by the alternate direction multiplier method (ADMM), which creates two new subproblems that are much easier to be computed. To accelerate the convergence speed of the proposed algorithm, we use an adaptive step size method to adjust the step parameter according to the residual of the dual problem at every iterate. We show the parallelization of the proposed algorithm and implement it on CUDA with block storage for the structured matrix, which is shown to be up to two orders of magnitude faster than the CPU implementation on large-scale problems.

On a primal-dual Newton proximal method for convex quadratic programs

This paper introduces QPDO, a primal-dual method for convex quadratic programs which builds upon and weaves together the proximal point algorithm and a damped semismooth Newton method. The outer proximal regularization yields a numerically stable method, and we interpret the proximal operator as the unconstrained minimization of the primal-dual proximal augmented Lagrangian function. This allows the inner Newton scheme to exploit sparse symmetric linear solvers and multi-rank factorization updates. Moreover, the linear systems are always solvable independently from the problem data and exact linesearch can be performed. The proposed method can handle degenerate problems, provides a mechanism for infeasibility detection, and can exploit warm starting, while requiring only convexity. We present details of our open-source C implementation and report on numerical results against state-of-the-art solvers. QPDO proves to be a simple, robust, and efficient numerical method for convex quadratic programming.

Solving the TSP by the AALHNN algorithm.

It is prone to get stuck in a local minimum when solving the Traveling Salesman Problem (TSP) by the traditional Hopfield neural network (HNN) and hard to converge to an efficient solution, resulting from the defect of the penalty method used by the HNN. In order to mend this defect, an accelerated augmented Lagrangian Hopfield neural network (AALHNN) algorithm was proposed in this paper. This algorithm gets out of the dilemma of penalty method by Lagrangian multiplier method, ensuring that the solution to the TSP is undoubtedly efficient. The second order factor added in the algorithm stabilizes the neural network dynamic model of the problem, thus improving the efficiency of solution. In this paper, when solving the TSP by AALHNN, some changes were made to the TSP models of Hopfield and Tank. Say, constraints of TSP are multiplied by Lagrange multipliers and augmented Lagrange multipliers respectively, The augmented Lagrange function composed of path length function can ensure robust convergence and escape from the local minimum trap. The Lagrange multipliers are updated by using nesterov acceleration technique. In addition, it was theoretically proved that the extremum obtained by this improved algorithm is the optimal solution of the initial problem and the approximate optimal solution of the TSP was successfully obtained several times in the simulation experiment. Compared with the traditional HNN, this method can ensure that it is effective for TSP solution and the solution to the TSP obtained is better.

Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Function

In this paper, we focus on the decentralized optimization problem over the Stiefel manifold, which is defined on a connected network of <inline-formula><tex-math notation="LaTeX">$d$</tex-math></inline-formula> agents. The objective is an average of <inline-formula><tex-math notation="LaTeX">$d$</tex-math></inline-formula> local functions, and each function is privately held by an agent and encodes its data. The agents can only communicate with their neighbors in a collaborative effort to solve this problem. In existing methods, multiple rounds of communications are required to guarantee the convergence, giving rise to high communication costs. In contrast, this paper proposes a decentralized algorithm, called DESTINY, which only invokes a single round of communications per iteration. DESTINY combines gradient tracking techniques with a novel approximate augmented Lagrangian function. The global convergence to stationary points is rigorously established. Comprehensive numerical experiments demonstrate that DESTINY has a strong potential to deliver a cutting-edge performance in solving a variety of testing problems.

AG-LRTR: An Adaptive and Generic Low-Rank Tensor-Based Recovery for IIoT Network Traffic Factors Denoising

The ubiquitous 5G-enable industrial Internet of Things interconnects a great number of intelligent sensors and actors. Network management becomes challenging due to massive traffic data generated by industrial equipment. However, the conventional single traffic factor is insufficient for the increasingly complicated network engineering tasks due to the poor representation capability. Besides, the insecure equipment with open communication access easily brings irregular network fluctuations to network traffic which interferes with the primary traffic factor. The simple and interfered traffic factor decreases the network management efficiency and misleads the operators. Motivated by that, we construct a comprehensive tensor model representing multi-dimension traffic factors to describe the network traffic beneficial characteristics. Meanwhile, an adaptive and generic low-rank tensor recovery (AG-LRTR) algorithm in the tensor singular value decomposition (t-SVD) framework is proposed for denoising. For effective tensor recovery, the alternating direction method of multipliers is employed to theoretically solve the partial augmented Lagrangian function of our objective with a closed-form solution. Numerical experiments on both synthetic data and real-world traffic data in IIoT validate that our proposed algorithm outperforms other state-of-the-art of tensor recovery algorithms.

Privacy-Preserved Distributed Learning With Zeroth-Order Optimization

We develop a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available and data is distributed over a multi-agent network. We employ a zeroth-order method to minimize the associated augmented Lagrangian function in the primal domain using the alternating direction method of multipliers (ADMM). We show that the proposed algorithm, named distributed zeroth-order ADMM (D-ZOA), has intrinsic privacy-preserving properties. Most existing privacy-preserving distributed optimization/estimation algorithms exploit some perturbation mechanism to preserve privacy, which comes at the cost of reduced accuracy. Contrarily, by analyzing the inherent randomness due to the use of a zeroth-order method, we show that D-ZOA is intrinsically endowed with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(\epsilon,\delta)-$ </tex-math></inline-formula> differential privacy. In addition, we employ the moments accountant method to show that the total privacy leakage of D-ZOA grows sublinearly with the number of ADMM iterations. D-ZOA outperforms the existing differentially-private approaches in terms of accuracy while yielding similar privacy guarantee. We prove that D-ZOA reaches a neighborhood of the optimal solution whose size depends on the privacy parameter. The convergence analysis also reveals a practically important trade-off between privacy and accuracy. Simulation results verify the desirable privacy-preserving properties of D-ZOA and its superiority over the state-of-the-art algorithms as well as its network-wide convergence.

A DAG-based cloud-fog layer architecture for distributed energy management in smart power grids in the presence of PHEVs

• New framework based on the directed acyclic graph and distributed multi-layer cloud-fog computing. • Finding a solution for the optimal energy management of the smart grids. • Considering a machine learning algorithm to solve augmented Lagrangian function in each agent. In this paper, a new framework based on the directed acyclic graph (DAG) and distributed multi-layer cloud-fog computing to find the optimal energy management of the smart grids, considering high penetration of plug-in hybrid electric vehicles (PHEVs). The presented distributed structure lets neighboring agents make a consensus together. The uncertainties have been modeled according to the Monte Carlo simulations, due to wide usages of diverse renewable energy resources such as photovoltaic panels and wind turbines. Three diverse charging schemes have been considered in the smart grid test system which contains controlled, uncontrolled and smart chargings. The Whale Optimization Algorithm (WOA) has been used to solve the augmented Lagrangian function in each agent. The simulation results are shown that the suggested scheme is effective.

Robust and stochastic compliance-based topology optimization with finitely many loading scenarios

In this paper, the problem of load uncertainty in compliance problems is addressed where the uncertainty is described in the form of a set of finitely many loading scenarios. Computationally, more efficient methods are proposed to exactly evaluate and differentiate: (1) the mean compliance or (2) any scalar-valued function of the individual load compliances such as the weighted sum of the mean and standard deviation. The computational time complexities of all the proposed algorithms are analyzed, compared with the naive approaches and then experimentally verified. Finally, a mean compliance minimization problem, a risk-averse compliance minimization problem, and a maximum compliance-constrained problem are solved to showcase the efficacy of the proposed algorithms. The maximum compliance-constrained problem is solved using the augmented Lagrangian method and the method proposed for handling scalar-valued functions of the load compliances, where the scalar-valued function is the augmented Lagrangian function.

Magnetic inversion to recover the subsurface block structures based onL1 norm and total variation regularization

SUMMARYThis paper presents a new sparse inversion method based on L1 norm regularization for 3-D magnetic data. In isolation, L1 norm regularization yields model elements which are unconstrained by the input data to be exactly zero, leading to a sparse model with compact and focused structure. Here, we complement the L1 norm with a penalty minimizing total variation, the L1 norm of the model gradients; it is expected that the sharp boundaries of the subsurface structure are not compromised by incorporating this penalty. Although this penalty is widely used in the geophysical inversion studies, it is often replaced by an alternative quadratic penalty to ease solution of the penalized inversion problem; in this study, the original definition of the total variation, that is form of the L1 norm of the model gradients, is used. To solve the problem with this combined penalty of L1 norm and total variation, this study introduces alternative direction method of multipliers, which is a primal-dual optimization algorithm that solves convex penalized problems based on the optimization of an augmented Lagrange function. To improve the computational efficiency of the algorithm to make this method applicable to large-scale magnetic inverse problems, this study applies matrix compression using the wavelet transform and the preconditioned conjugate gradient method. The inversion method is applied to both synthetic tests and real data, the synthetic tests demonstrate that, when subsurface structure is blocky, it can be reproduced almost perfectly.

Residual Life Prediction of Lithium Battery Based on Small Sample Data Sets

In the recycling process of lithium-ion battery, the poor external environment will lead to the degradation of battery performance, so it is necessary to predict the residual life of lithiumion battery. However, the aging cycle of lithium-ion battery is long, so it is difficult to obtain sufficient aging data in a short time. Aiming at the problem of low prediction accuracy of lithium battery residual life under the condition of small samples, a BP neural network modelling algorithm fused with expert knowledge is proposed. Firstly, indirect health factors were extracted from the aging data of lithium battery in NASA PCOE laboratory. Secondly, the initial weight and threshold of BP neural network were optimized by genetic algorithm, and the initial multiplier value of the augmented Lagrange function was set according to Spearman correlation coefficient. Finally, the expert knowledge is integrated into the training process of BP neural network through the augmented Lagrange multiplier method. Simulation results show that the proposed algorithm has higher prediction accuracy than the traditional BP neural network under the condition of small samples.

Decentralized Voltage Optimization Based on the Auxiliary Problem Principle in Distribution Networks with DERs

This paper addresses the problem of optimizing the voltage profile of radially-operated distribution systems by acting on the active and reactive powers provided by distributed energy resources (DERs). A novel voltage optimization procedure is proposed by adopting a decentralized control strategy. To this aim, a centralized voltage optimization problem (VOP), minimizing the distance of all the nodal voltages from their reference values, is firstly formulated as a strictly-convex quadratic program. Then, the centralized VOP is rewritten by partitioning the network into voltage control zones (VCZs) with pilot nodes. To overcome the lack of strictly convexity determined by the reduction to the pilot nodes, the dual centralized VOP working on the augmented Lagrangian function is reformulated and iteratively solved by the method of multipliers. Finally, a fully-distributed VOP solution is obtained by applying a distributed algorithm based on the auxiliary problem principle, which allows for solving in each VCZ a quadratic programming problem of small dimension and to drive the VCZ solutions toward the overall optimum by an iterative coordination process that requires to exchange among the VCZs only scalar values. The effectiveness and feasibility of the proposed method have been demonstrated via numerical tests on the IEEE 123-bus system.

Distributed Nonconvex Optimization for Energy Efficiency in Mobile Ad Hoc Networks

This article proposes a distributed nonconvex optimization algorithm for energy efficiency (EE) in mobile ad hoc networks (MANETs). The proposed model for EE in MANETs can be classified into the category of nonconvex optimization problems. There are various distributed algorithms for solving nonconvex optimization problems. But, they are associated with some flaws such as complexity of implementation and low convergence rate. In this regard, in addition to the simplicity in implementation, the optimal point of the proposed model is a saddle point. Using augmented Lagrangian function with continuous-time optimization dynamics algorithm, a dynamic system will be established where its equilibrium points are Karush–Kuhn–Tucker points of the proposed model. Also, it will be shown that these points can be locally asymptotically stable under some conditions. Additionally, the proposed analytic method proves that the rate of convergence increases with an increasing penalty coefficient. To validate the performance of the presented algorithm, the problem of EE optimization for a sample mobile ad hoc network is simulated in MATLAB environment.

Sparse intrinsic decomposition and applications

This paper proposes an intrinsic decomposition method from a single RGB-D image. To remedy the highly ill-conditioned problem, the reflectance component is regularized by a sparsity term, which is weighted by a bilateral kernel to exploit non-local structural correlation. As shading images are piece-wise smooth and have sparse gradient fields, the sparse-induced ℓ1-norm is used to regularize the finite difference of the direct irradiance component, which is the most dominant sub-component of shading and describes the light directly received by the surfaces of the objects from the light source. To derive an efficient algorithm, the proposed model is transformed into an unconstrained minimization of the augmented Lagrangian function, which is then optimized via the alternating direction method. The stability of the proposed method with respect to parameter perturbation and its robustness to noise are investigated by experiments. Quantitative and qualitative evaluation demonstrates that our method has better performance than state-of-the-art methods. Our method can also achieve intrinsic decomposition from a single color image by integrating existed depth estimation methods. We also present a depth refinement method based on our intrinsic decomposition method, which obtains more geometry details without texture artifacts. Other application, e.g., texture editing, also demonstrates the effectiveness of our method.

Duality in nonconvex vector optimization

In this paper, duality relations in nonconvex vector optimization are studied. An augmented Lagrangian function associated with the primal problem is introduced and efficient solutions to the given vector optimization problem, are characterized in terms of saddle points of this Lagrangian. The dual problem to the given primal one, is constructed with the help of the augmented Lagrangian introduced and weak and strong duality theorems are proved. Illustrative examples for duality relations are provided.

Fourier ptychographic reconstruction based on augmented Lagrangian method and sparse approximations for phase and magnitude

Fourier ptychography microscopy provides a large field of view and high-resolution imaging by simultaneously recovering intensity and phase distributions. However, in real setups, the process of capturing large numbers of low-resolution images will inevitably suffer from imaging noise, which could seriously distort the results recovered using the conventional Fourier ptychography approach. To suppress the effects of imaging noise optimally, a novel, to the best of our knowledge, iterative algorithm is proposed. This algorithm consists of two objective functions; one is based on the augmented Lagrangian function for the inverse computation, and its solution is found by utilizing the alternating direction multiplier method; the other is the separate sparse model built for amplitude and absolute phase image; the filtering process is accomplished by exploiting the block-matching 3D frames. In combination with the Nash equilibrium balancing theory, the proposed algorithm is realized by alternately optimizing the two objective functions. The simulated and experimental results demonstrate that the proposed algorithm is robust to noise and is capable of reconstructing complete and good contrast amplitude and phase images.

A Distributed Optimal Scheduling Method Based on Microgrid Cluster of Plug and Play

This paper proposes a distributed optimization method to solve the problems of centralized optimization and centralized management of a microgrid. Also, the distributed optimization solution method upgrade to a process of distributed iterative solution and optimization, which can solve the distributed optimization problem of a large microgrid cluster. According to iterative calculation, accord the augmented Lagrange function supports the centralized optimization problem divided into corresponding subproblems, and the penalty factors of interconnected variables considered to adjust the consumption of local resources, to make microgrid cluster (MGC) more flexible. In particular, the framework of multi-verse consistency and model predictive control (MPC) can help the global optimization reach the optimal quickly. In this paper, the simulation is solved by Gurobi commercial solver in MATLAB. The results show that the proposed method needs only a few iterations to achieve global optimization, and the effectiveness of plug and play proved.

Monotone Fuzzy Rule Interpolation for Practical Modeling of the Zero-Order TSK Fuzzy Inference System

Formulating a generalized monotone fuzzy rule interpolation (MFRI) model is difficult. A complete and monotone fuzzy rule-base is essential for devising a monotone zero-order Takagi–Sugeno–Kang (TSK) fuzzy inference system (FIS) model. However, such a complete and monotone fuzzy rule-base is not always available in practice. In this article, we develop an MFRI modeling scheme for generating a monotone zero-order TSK FIS from a monotone and incomplete fuzzy rule-base. In our proposal, a monotone-ordered fuzzy rule-base that consists of the available fuzzy rules from a monotone and incomplete fuzzy rule-base and those derived from the MFRI reasoning is formed. We outline three important properties that the MFRI's deduced fuzzy rules should satisfy to ensure a monotone-ordered fuzzy rule-base. A Lagrangian function for the MFRI scheme, together with its Karush–Kuhn–Tucker optimality conditions, is formulated and analyzed. The key idea is to impose constraints that guide the MFRI inference outcomes. An iterative MFRI algorithm that adopts an augmented Lagrangian function is devised. The proposed MFRI algorithm aims to achieve an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\boldsymbol{\varepsilon }}$</tex-math></inline-formula> -optimality condition and to produce an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\boldsymbol{\varepsilon }}$</tex-math></inline-formula> -optimal solution, which is geared for practical applications. We apply the MFRI algorithm to a failure mode and effect analysis case study and a tanker ship heading regulation problem. The results indicate the effectiveness of MFRI for generating monotone TSK FRI models in tackling practical problems.

A Neurodynamic Optimization Approach for L1 Minimization with Application to Compressed Image Reconstruction

This paper presents a neurodynamic optimization approach for l1 minimization based on an augmented Lagrangian function. By using the threshold function in locally competitive algorithm (LCA), subgradient at a nondifferential point is equivalently replaced with the difference of the neuronal state and its mapping. The efficacy of the proposed approach is substantiated by reconstructing three compressed images.