Nonsmooth Objective Research Articles

Two inertial proximal coordinate algorithms for a family of nonsmooth and nonconvex optimization problems

The inertial proximal method is extended to minimize the sum of a series of separable nonconvex and possibly nonsmooth objective functions and a smooth nonseparable function (possibly nonconvex). Here, we propose two new algorithms. The first one is an inertial proximal coordinate subgradient algorithm, which updates the variables by employing the proximal subgradients of each separable function at the current point. The second one is an inertial proximal block coordinate method, which updates the variables by using the subgradients of the separable functions at the partially updated points. Global convergence is guaranteed under the Kurdyka–Łojasiewicz (KŁ) property and some additional mild assumptions. Convergence rate is derived based on the Łojasiewicz exponent. Two numerical examples are given to illustrate the effectiveness of the algorithms.

Distributed continuous-time algorithm for nonsmooth aggregative optimization over weight-unbalanced digraphs

This paper studies the problem of distributed continuous-time aggregative optimization with set constraints under a weight-unbalanced digraph, where the nonsmooth objective function of each agent relies both on its own decision and on the aggregation of all agents’ decisions. To eliminate the impact of unbalanced digraphs, a consensus-based estimator that tracks the aggregation information is designed through a gradient rescaling technique. Considering that cost functions are nondifferentiable in many scenarios, such as electric power management that takes price caps into account, a novel distributed continuous-time optimization algorithm via generalized gradient is presented in a two-time scale. Moreover, the convergence of the algorithm is established through nonsmooth analysis and singular perturbation theory. Compared to the existing results, which depend on undirected graphs, the proposed strategy is applicable to general digraphs, which may be weight-unbalanced. Further, the assumption on the differentiability of objective functions is relaxed. Finally, two numerical examples are provided to verify the findings.

Shaped pattern synthesis for hybrid analog–digital arrays via manifold optimization-enabled block coordinate descent

Hybrid analog–digital (HAD) architecture is a promising means to realize large-scale arrays owing to the judicious trade-off between system performance and hardware complexity. This paper investigates the beampattern shaping of HAD arrays through minimizing the beampattern matching error between desired and actual patterns. Due to the nonconvex constraints on analog and digital weights and the nonconvex nonsmooth objective function, the resultant codesign is NP-hard. To address this issue, we create an efficient algorithm by leveraging block coordinate descent (BCD) and Riemannian manifold optimization. We first equivalently represent the objective function as a smooth bi-quadratic function by introducing a uni-modulus auxiliary variable. Subsequently, we alternatingly optimize the scaling factor, digital weights, analog weights and auxiliary variable under the BCD framework, where the simultaneous update of analog weights and auxiliary variable is recast as uni-modular constrained quadratic programming which is efficiently solved by Riemannian Newton method, and the digital weights have a global optimal solution via Lagrangian multiplier method. We also derive an explicit convergence condition of this algorithm. Numerical results demonstrate that the proposed algorithm has superior performance and faster convergence speed than alternative algorithms, and produces nearly the same mainlobe gain as fully digital arrays with significantly fewer radio-frequency chains.

A novel improved harbor seal whiskers algorithm for solving hybrid dynamic economic environmental dispatch considering uncertainty of renewable energy generation

The originality of this work is introducing a novel Improved Harbor Seal Whiskers Algorithm (IHSWA) as an innovative and improved optimizer for solving complex hybrid dynamic economic-environmental dispatch (HDEED) considering uncertainty of renewable energy generation (REG) and various system constraints. IHSWA is implemented by hybridizing opposition-based algorithm (OBA) and local search algorithm (LSA) approaches to enhance the algorithm's search efficiency and overcome the challenges of other counterpart approaches. IHSWA has an optimal balancing feature between searching modes (exploring and exploiting modes). This feature makes the proposed algorithms more advanced than their counterpart optimizing techniques. The HDEED is modeled using a non-convex and non-smooth objective function, considering various equality and inequality constraints such as ramp rate bounds (RRLs), valve-point effects (VPE), production limits (PL), spinning reserves (SRLs), prohibited operating zones (POZs), and power balance (PB). The proposed approach is verified on a hybrid modified IEEE 57-bus 7-unit power system incorporating solar and wind energy generation. The proposed approach is compared with various modern optimization techniques, such as the traditional Harbor Seal Whiskers Algorithm (HSWA), the Lemurs Optimization Algorithm (LOA), the Nutcracker Optimization Algorithm (NOA), and the Artificial Hummingbird Algorithm (AHA). The incorporation of REG sources produces a significant reduction in operative costs and a remarkable decline in emissions. The convergence rate of the proposed approach indicates that the novel hybrid algorithm converges rapidly towards the optimal solution, achieving the best global optimum solution. The ANOVA test and the Tukey test are used to analyze the results statistically. The IHSWA approach shows superior advanced features with a best fuel cost of 1640.5890 $ and a minimum std. value of 1708.4895. While for emission objective minimizing, IHSWA attained a minimum emission cost and std. value of 1080.1629 kg and 1777.0770, respectively. The statistical analysis of the results approves the stability, reliability, and consistency of the proposed strategy with significant convergence. The appreciated contribution of this work is introducing a novel hybrid approach for solving HDEED efficiently, incorporating REG, and considering various system constraints. IHSWA has advanced features over its counterpart approaches. The contribution holds specific importance for the economic development of countries. It creates the path for increased self-sufficiency by reducing their need for fossil fuels for energy production. The study emphasizes the transition from being reliant on fossil fuels to the development of sustainable urban economies.

AN-SPS: adaptive sample size nonmonotone line search spectral projected subgradient method for convex constrained optimization problems

We consider convex optimization problems with a possibly nonsmooth objective function in the form of a mathematical expectation. The proposed framework (AN-SPS) employs Sample Average Approximations (SAA) to approximate the objective function, which is either unavailable or too costly to compute. The sample size is chosen in an adaptive manner, which eventually pushes the SAA error to zero almost surely (a.s.). The search direction is based on a scaled subgradient and a spectral coefficient, both related to the SAA function. The step size is obtained via a nonmonotone line search over a predefined interval, which yields a theoretically sound and practically efficient algorithm. The method retains feasibility by projecting the resulting points onto a feasible set. The a.s. convergence of AN-SPS method is proved without the assumption of a bounded feasible set or bounded iterates. Preliminary numerical results on Hinge loss problems reveal the advantages of the proposed adaptive scheme. In addition, a study of different nonmonotone line search strategies in combination with different spectral coefficients within AN-SPS framework is also conducted, yielding some hints for future work.

Objective penalty function method for nonlinear programming with inequality constraints

<p>This paper presents a novel nonsmooth objective penalty function for inequality constrained optimization problems. A modified flattened aggregate function, which is a smooth approximation of the max-value function, is discussed. Then, the smooth objective penalty function that contains the flattened aggregate function is proposed, and the exactness of the new function is studied. Based on this, an objective penalty function algorithm is proposed and its convergence is proven under mild conditions. Because of the flattened aggregate function, the gradient computation can usually be greatly reduced for problems with many constraints. Numerical experiments are included to illustrate the efficiency of the new algorithm through a series of numerical examples, especially for solving problems with many constraints.</p>

A constraint dissolving approach for nonsmooth optimization over the Stiefel manifold

Abstract This paper focuses on the minimization of a possibly nonsmooth objective function over the Stiefel manifold. The existing approaches either lack efficiency or can only tackle prox-friendly objective functions. We propose a constraint dissolving function named NCDF and show that it has the same first-order stationary points and local minimizers as the original problem in a neighborhood of the Stiefel manifold. Furthermore, we show that the Clarke subdifferential of NCDF is easy to achieve from the Clarke subdifferential of the objective function. Therefore, various existing approaches for unconstrained nonsmooth optimization can be directly applied to nonsmooth optimization problems over the Stiefel manifold. We propose a framework for developing subgradient-based methods and establishing their convergence properties based on prior works. Furthermore, based on our proposed framework, we can develop efficient approaches for optimization over the Stiefel manifold. Preliminary numerical experiments further highlight that the proposed constraint dissolving approach yields efficient and direct implementations of various unconstrained approaches to nonsmooth optimization problems over the Stiefel manifold.

Improved Pelican optimization algorithm for solving load dispatch problems

The Pelican Optimization Algorithm (POA) is a newly developed algorithm inspired by the hunting behavior of pelicans. Despite its fast convergence rate, it suffers from premature convergence, the imbalance between exploration and exploitation, and lack of population diversity. In this work, an improved POA is proposed to attenuate these shortcomings. IPOA benefits from three motion strategies and predefined knowledge-sharing factors that better describe the stochastic hunting behavior of pelicans, as well as a modified dimension-learning-based hunting (DHL) behavior to retain diversity. To test the effectiveness of these improvements, it was used to solve 23 benchmark functions, including unimodal, multimodal, and 6 composite functions (CEC 2017). To evaluate performance, it was applied to solve economic and combined economic emission load dispatch problems that play a critical role in real-world power system planning and operation while considering environmental impacts. This experiment includes 6, 10, 11, 40, 140, 160, and 320 generating units with nonconvex and non-smooth objective functions. The comparison is performed for benchmark functions and the optimal dispatch problems. The results confirm the competitive and almost superior performance of the IPOA in several cases, which proves the applicability and efficiency of the proposed approach in solving real-world problems.

A robust sparse identification method for nonlinear dynamic systems affected by non-stationary noise.

In the field of science and engineering, identifying the nonlinear dynamics of systems from data is a significant yet challenging task. In practice, the collected data are often contaminated by noise, which often severely reduce the accuracy of the identification results. To address the issue of inaccurate identification induced by non-stationary noise in data, this paper proposes a method called weighted ℓ1-regularized and insensitive loss function-based sparse identification of dynamics. Specifically, the robust identification problem is formulated using a sparse identification mathematical model that takes into account the presence of non-stationary noise in a quantitative manner. Then, a novel weighted ℓ1-regularized and insensitive loss function is proposed to account for the nature of non-stationary noise. Compared to traditional loss functions like least squares and least absolute deviation, the proposed method can mitigate the adverse effects of non-stationary noise and better promote the sparsity of results, thereby enhancing the accuracy of identification. Third, to overcome the non-smooth nature of the objective function induced by the inclusion of loss and regularization terms, a smooth approximation of the non-smooth objective function is presented, and the alternating direction multiplier method is utilized to develop an efficient optimization algorithm. Finally, the robustness of the proposed method is verified by extensive experiments under different types of nonlinear dynamical systems. Compared to some state-of-the-art methods, the proposed method achieves better identification accuracy.

On the Global Convergence of Particle Swarm Optimization Methods

In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a continuous-time formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution, which acts as Lyapunov function of the dynamics. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

An Extensive Study Using the Beetle Swarm Method to Optimize Single and Multiple Objectives of Various Optimal Power Flow Problems

An electric energy generation system, under the economic operation mode, is an imperative mission in the power system function. This article deals with the use of beetle swarm optimization algorithm (BSOA), for optimal power flow (OPF) solution, in an effective approach. BSOA is a competent optimization technique, to handle multimodal, nonlinear, and nondifferentiable objective functions. The proposed OPF is modeled by numerous objective functions, formulations with constraints, examined with thirty-one different cases, on the three distinguished test systems (IEEE 30, 57, and 118-bus), using single and weighted sum multiobjectives. Six new multiobjective cases are also studied. The control variables, such as real generation of power, tap setting ratio of transformers, bus voltages magnitudes, and the values of shunt capacitor, are also optimized. Potency and robustness of this proposed method were investigated and evaluated with more recent findings reported in the literature. This extensive study revealed the preeminence of the presented technique, applied to OPF problem, with intricate and nonsmooth objective functions.

Formal convergence analysis on deterministic [formula omitted]-regularization based mini-batch learning for RBF networks

Conventional convergence analysis on mini-batch learning is usually based on the stochastic gradient concept, in which we assume that the training data are presented in a random order. Also, some convergence results require that the learning rate should decrease with the number of training cycles, and that the objective function is a smooth function. Practically speaking, a deterministic presentation scheme with a fixed learning rate is more preferable. Hence, there is a gap between theoretical results and actual implementation. This paper aims at filling the gap. We use the radial basis function (RBF) model for nonlinear regression problems as an example to analyze the convergence properties of mini-batch learning. This paper considers a nonsmooth objective function, which consists of three terms. The coexistence of these three terms is able to handle a number of situations. The first term is a conventional training set error. The second term is a quadratic term which is used to suppress the effect of imperfections in the implementation. The last term is an ℓ1-norm term which is used to select important RBF nodes for the resultant network. Note that the ℓ1-norm term is a nonsmooth function. Although a nonsmooth ℓ1-norm is included and the mini-batch algorithm is deterministic, we are still able to derive the convergence properties, including the sufficient conditions for convergence and range of learning rate. With our results, we have a better theoretical understanding on the behaviour of mini-batch learning and obtain some guidelines on choosing the learning rate. The analysis results can be extended to other flat structural neural network models and other objective functions, which are with quadratic terms and ℓ1-norm.

Set-Limited Functions and Polynomial-Time Interior-Point Methods

In this paper, we revisit some elements of the theory of self-concordant functions. We replace the notion of self-concordant barrier by a new notion of set-limited function, which forms a wider class. We show that the proper set-limited functions ensure polynomial time complexity of the corresponding path-following method (PFM). Our new PFM follows a deviated path, which connects an arbitrary feasible point with the solution of the problem. We present some applications of our approach to the problems of unconstrained optimization, for which it ensures a global linear rate of convergence even in for nonsmooth objective function.

Distributed neurodynamic algorithms for collaborative energy management in energy internet considering time-varying factors

This paper investigates the energy management problem of the energy Internet under time-varying conditions . In the context of coupled multi-energy networks, the energy Internet is considered to be composed of multiple energy bodies and requires collaborative planning of multiple energy networks. A model for distributed energy management with a non-smooth cost function and line congestion constraints is proposed, with the goal of reducing overall operating costs and improving customer benefits while considering load as a time-varying factor. Then, a neurodynamic time-varying algorithm for addressing the energy management problem executed in a fully distributed manner is proposed. On the one hand, the predictive effect of the differential feedback term is exploited and embedded in the implementation of the proposed algorithm, thus speeding up the convergence. On the other hand, the algorithm is executed in a distributed manner, and only limited information is exchanged among the agents to complete the optimal operation locally, thus reducing the communication burden and ensuring privacy and robustness. Finally, theoretical proofs guarantee the stability of the proposed algorithm, and simulation experiments illustrate the effectiveness and robustness of the proposed algorithm. • In this paper, a model for EI in a time-varying load scenario is proposed for optimal energy allocation among EBs integrating renewable energy, flexible loads, and multi-energy coupling, while trading energy. • A neurodynamic algorithm implemented in a fully distributed manner that enjoys hardware implementation and parallel computation is proposed in this paper. • The algorithm can effectively tackle EMPs with non-smooth objective functions and constraints under time-varying conditions.

On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

<p style='text-indent:20px;'>Second-order continuous-time dissipative dynamical systems with viscous and Hessian driven damping have inspired effective first-order algorithms for solving convex optimization problems. While preserving the fast convergence properties of the Nesterov-type acceleration, the Hessian driven damping makes it possible to significantly attenuate the oscillations. To study the stability of these algorithms with respect to perturbations, we analyze the behaviour of the corresponding continuous systems when the gradient computation is subject to exogenous additive errors. We provide a quantitative analysis of the asymptotic behaviour of two types of systems, those with implicit and explicit Hessian driven damping. We consider convex, strongly convex, and non-smooth objective functions defined on a real Hilbert space and show that, depending on the formulation, different integrability conditions on the perturbations are sufficient to maintain the convergence rates of the systems. We highlight the differences between the implicit and explicit Hessian damping, and in particular point out that the assumptions on the objective and perturbations needed in the implicit case are more stringent than in the explicit case.</p>

Exploiting task relationships for Alzheimer’s disease cognitive score prediction via multi-task learning

Alzheimer’s disease (AD) is highly prevalent and a significant cause of dementia and death in elderly individuals. Motivated by breakthroughs of multi-task learning (MTL), efforts have been made to extend MTL to improve the Alzheimer’s disease cognitive score prediction by exploiting structure correlation. Though important and well-studied, three key aspects are yet to be fully handled in an unified framework: (i) appropriately modeling the inherent task relationship; (ii) fully exploiting the task relatedness by considering the underlying feature structure. (iii) automatically determining the weight of each task. To this end, we present the Bi-Graph guided self-Paced Multi-Task Feature Learning (BGP-MTFL) framework for exploring the relationship among multiple tasks to improve overall learning performance of cognitive score prediction. The framework consists of the two correlation regularization for features and tasks, ℓ2,1 regularization and self-paced learning scheme. Moreover, we design an efficient optimization method to solve the non-smooth objective function of our approach based on the Alternating Direction Method of Multipliers (ADMM) combined with accelerated proximal gradient (APG). The proposed model is comprehensively evaluated on the Alzheimer’s disease neuroimaging initiative (ADNI) datasets. Overall, the proposed algorithm achieves an nMSE (normalized Mean Squared Error) of 3.923 and an wR (weighted R-value) of 0.416 for predicting eighteen cognitive scores, respectively. The empirical study demonstrates that the proposed BGP-MTFL model outperforms the state-of-the-art AD prediction approaches and enables identifying more stable biomarkers.

An Adaptive Continuous-Time Algorithm for Nonsmooth Convex Resource Allocation Optimization

This article develops a novel continuous-time algorithm based on the idea of adaptive strategy for solving a resource allocation optimization with nonsmooth objective functions and constraints over multiagent network. It is proved that the state solution is globally bounded and finally converges to an optimal solution to the nonsmooth convex resource allocation problem. Compared with the existing algorithms, the strong/strict convexity of the objective function is relaxed and only convexity is required. Moreover, by employing an exact penalty approach for the distributed optimization, the primal-dual variables is avoided to introduce. Therefore, the proposed algorithm has a simple structure with low dimensionality of state variables. To show the effectiveness and practicability of the presented algorithm, a numerical example and an application in power system are presented.

A semismooth Newton based augmented Lagrangian method for nonsmooth optimization on matrix manifolds

This paper is devoted to studying an augmented Lagrangian method for solving a class of manifold optimization problems, which have nonsmooth objective functions and nonlinear constraints. Under the constant positive linear dependence condition on manifolds, we show that the proposed method converges to a stationary point of the nonsmooth manifold optimization problem. Moreover, we propose a globalized semismooth Newton method to solve the augmented Lagrangian subproblem on manifolds efficiently. The local superlinear convergence of the manifold semismooth Newton method is also established under some suitable conditions. We also prove that the semismoothness on submanifolds can be inherited from that in the ambient manifold. Finally, numerical experiments on compressed modes and (constrained) sparse principal component analysis illustrate the advantages of the proposed method.

Spectral projected subgradient method for nonsmooth convex optimization problems

We consider constrained optimization problems with a nonsmooth objective function in the form of mathematical expectation. The Sample Average Approximation (SAA) is used to estimate the objective function and variable sample size strategy is employed. The proposed algorithm combines an SAA subgradient with the spectral coefficient in order to provide a suitable direction which improves the performance of the first order method as shown by numerical results. The step sizes are chosen from the predefined interval and the almost sure convergence of the method is proved under the standard assumptions in stochastic environment. To enhance the performance of the proposed algorithm, we further specify the choice of the step size by introducing an Armijo-like procedure adapted to this framework. Considering the computational cost on machine learning problems, we conclude that the line search improves the performance significantly. Numerical experiments conducted on finite sum problems also reveal that the variable sample strategy outperforms the full sample approach.

Distributed online adaptive subgradient optimization with dynamic bound of learning rate over time‐varying networks

Adaptive online optimization algorithms, such as Adam, RMSprop, and AdaBound, have recently been tremendously popular as they have been widely applied to address the issues in the field of deep learning. Despite their prevalence and prosperity, however, it is rare to investigate the distributed versions of these adaptive online algorithms. To fill the gap, a distributed online adaptive subgradient learning algorithm over time-varying networks, called DAdaxBound, which exponentially accumulates long-term past gradient information and possesses dynamic bounds of learning rates under learning rate clipping is developed. Then, the dynamic regret bound of DAdaxBound on convex and potentially nonsmooth objective functions is theoretically analysed. Finally, numerical experiments are carried out to assess the effectiveness of DAdaxBound on different datasets. The experimental results demonstrate that DAdaxBound compares favourably to other competing distributed online optimization algorithms.