Subgradient Method Research Articles

A modified subgradient extragradient algorithm-type for solving quasimonotone variational inequality problems with applications

In this article, we introduce an inertial-type algorithm that combines the extragradient subgradient method, the projection contraction method, and the viscosity method. The proposed method is used for solving quasimonotone variational inequality problems in infinite dimensional real Hilbert spaces such that it does not depend on the Lipschitz constant of the cost operator. Further, we prove the strong convergence results of the new algorithm. Our strong convergence results are achieved without imposing strict conditions on the control parameters and inertial factor of our algorithm. We utilize our algorithm to solve some problems in applied sciences and engineering such as image restoration and optimal control. Some numerical experiments are carried out to support our theoretical results. Our numerical illustrations show that our new method is more efficient than many existing methods.

A Family of Multi-Step Subgradient Minimization Methods

For solving non-smooth multidimensional optimization problems, we present a family of relaxation subgradient methods (RSMs) with a built-in algorithm for finding the descent direction that forms an acute angle with all subgradients in the neighborhood of the current minimum. Minimizing the function along the opposite direction (with a minus sign) enables the algorithm to go beyond the neighborhood of the current minimum. The family of algorithms for finding the descent direction is based on solving systems of inequalities. The finite convergence of the algorithms on separable bounded sets is proved. Algorithms for solving systems of inequalities are used to organize the RSM family. On quadratic functions, the methods of the RSM family are equivalent to the conjugate gradient method (CGM). The methods are intended for solving high-dimensional problems and are studied theoretically and numerically. Examples of solving convex and non-convex smooth and non-smooth problems of large dimensions are given.

Control learning rate for autism facial detection via deep transfer learning.

Autism spectrum disorder (ASD) is a complex neurodevelopmental disorder that affects social interaction and communication. Early detection of ASD can significantly improve outcomes for individuals with the disorder, and there has been increasing interest in using machine learning techniques to aid in the diagnosis of ASD. One promising approach is the use of deep learning techniques, particularly convolutional neural networks (CNNs), to classify facial images as indicative of ASD or not. However, choosing a learning rate for optimizing the performance of these deep CNNs can be tedious and may not always result in optimal convergence. In this paper, we propose a novel approach called the control subgradient algorithm (CSA) for tackling ASD diagnosis based on facial images using deep CNNs. CSA is a variation of the subgradient method in which the learning rate is updated by a control step in each iteration of each epoch. We apply CSA to the popular DensNet-121 CNN model and evaluate its performance on a publicly available facial ASD dataset. Our results show that CSA is faster than the baseline method and improves the classification accuracy and loss compared to the baseline. We also demonstrate the effectiveness of using CSA with -regularization to further improve the performance of our deep CNN model.

Distributed primal-dual method on unbalanced digraphs with row stochasticity

This study investigates distributed convex optimisation that accounts for the inequality constraints over unbalanced directed graphs. In distributed optimisation, agents exchange information on a network to obtain an optimal solution when they only know their own cost function. We propose a distributed primal-dual subgradient method based on a row stochastic weight matrix that is associated with a communication network. In the proposed method, the normalised left eigenvector of the weight matrix is estimated by a consensus algorithm. Then, the subgradient of the Lagrange function is scaled by the estimated values of the left eigenvector to compensate for the imbalance of the information flow in the network. We show that the pair of estimations for the primal and dual problems converge to an optimal primal-dual solution. We also show the relation between the convergence rate of the proposed algorithm and the step-size rule. A numerical example confirms the validity of the proposed method.

Adaptive Subgradient Methods for Leadership And Development

Adaptive Subgradient Methods for Leadership And Development

A generalized inertial proximal alternating linearized minimization method for nonconvex nonsmooth problems

We propose a general inertial version of the proximal alternating linearized minimization (PALM) (denoted by NiPALM) for a class of nonconvex and nonsmooth minimization problems, whose objective function is the sum of a smooth function of the entire variables and two nonsmooth functions of each variable. NiPALM is general in the sense that it contains the popular PALM, the inertial PALM (iPALM) and Gauss-Seidel type inertial PALM (GiPALM) as special cases. Under mild assumptions, namely, the underlying functions satisfy the Kurdyka-Łojasiewicz (KL) property and some suitable conditions on the parameters, we prove that each bounded sequence generated by NiPALM globally converges to a critical point. We also apply NiPALM to nonnegative matrix factorization, sparse principal component analysis, and weighted low-rank matrix restoration problems. Comparing results with those by PALM, iPALM, and GiPALM demonstrate the robustness and effectiveness of the proposed algorithm.

Publisher Correction to: Lyapunov stability of the subgradient method with constant step size

Publisher Correction to: Lyapunov stability of the subgradient method with constant step size

Continuous Network Design Using Partial Linearized Subgradient Methods

Although demand management has shown to be a vital tool in managing congestion, many metropolitan planning organizations (MPOs) and departments of transportation (DOTs) still pursue network modification and capacity increase as a congestion relief and a mitigation measure. The network design problem (NDP) still has, therefore, an essential and significant role in shaping and sizing urban transportation networks. The literature has traditionally treated the NDP as a bi-level mathematical programming problem or a mathematical program with equilibrium constraints (MPEC). In the bi-level optimization setting, the problem is approached as a leader-follower problem in which the lower level is a user equilibrium (UE) assignment problem as the follower, and the upper level is the network sizing problem as the leader problem. NDP has long been known as a challenging problem, and many solution algorithms have been proposed to solve it. This study proposed an efficient solution algorithm for the continuous network design problem (CNDP). The solution algorithm has been shown empirically to solve the CNDP in a shorter time using partial linearized subgradient methods. The proposed method was applied to a small network that was traditionally used to evaluate the performance of solution algorithms versus Braess’s paradox. It was then applied to the Sioux Falls network as a well-known benchmark network to compare the results with previous studies. The proposed method has been shown to run much faster than all the previous studies reviewed in this paper with minimal degradation of accuracy (0.52% lower than the best solution).

Mini-batch stochastic subgradient for functional constrained optimization

In this paper, we consider finite sum composite optimization problems with many functional constraints. The objective function is expressed as a finite sum of two terms, one of which admits easy computation of (sub)gradients while the other is amenable to proximal evaluations. We assume a generalized bounded gradient condition on the objective which allows us to simultaneously tackle both smooth and nonsmooth problems. We also consider the cases of both with and without a quadratic functional growth property. Further, we assume that each constraint set is given as the level set of a convex but not necessarily a differentiable function. We reformulate the constrained finite sum problem into a stochastic optimization problem for which the stochastic subgradient projection method from Necoara and Singh [Stochastic subgradient projection methods for composite optimization with functional constraints; 2022 Journal of Machine Learning Research, 23, 1–35] specializes in a collection of mini-batch variants, with different mini-batch sizes for the objective function and functional constraints, respectively. More specifically, at each iteration, our algorithm takes a mini-batch stochastic proximal subgradient step aimed at minimizing the objective function and then a subsequent mini-batch subgradient projection step minimizing the feasibility violation. By specializing in different mini-batching strategies, we derive exact expressions for the stepsizes as a function of the mini-batch size and in some cases we also derive insightful stepsize-switching rules which describe when one should switch from a constant to a decreasing stepsize regime. We also prove sublinear convergence rates for the mini-batch subgradient projection algorithm which depend explicitly on the mini-batch sizes and on the properties of the objective function. Numerical results also show a better performance of our mini-batch scheme over its single-batch counterpart.

Charging station location problem for maximizing the space-time-electricity accessibility: A Lagrangian relaxation-based decomposition scheme

The driving range of electric vehicles limits the accessibility of their users. Charging infrastructures such as charging stations are essential to improve the accessibility of electric vehicles. Existing charging station location studies neglected accessibility-based indicators in their optimization models. For the charging station location problem, this paper proposes a novel objective function that maximizes the space–time-electricity accessibility of electric vehicles. Then we formulate an integer-programming model in the space–time-electricity network. A Lagrangian relaxation-based decomposition scheme is developed to solve this problem. The constraints that couple flow with location variables are dualized to the objective function, resulting in a set of independent routing subproblems and a knapsack subproblem. At each iteration, a primal heuristic utilizes the result of the intermediate knapsack subproblem to generate a feasible solution and Lagrangian multipliers are updated by the subgradient optimization method. The numerical experiments are conducted on three networks, showing that the proposed method achieves good integrality gaps.

Properties of the Quadratic Transformation of Dual Variables

We investigate a solution of a convex programming problem with a strongly convex objective function based on the dual approach. A dual optimization problem has constraints on the positivity of variables. We study the methods and properties of transformations of dual variables that enable us to obtain an unconstrained optimization problem. We investigate the previously known method of transforming the components of dual variables in the form of their modulus (modulus method). We show that in the case of using the modulus method, the degree of the degeneracy of the function increases as it approaches the optimal point. Taking into account the ambiguity of the gradient in the boundary regions of the sign change of the new dual function variables and the increase in the degree of the function degeneracy, we need to use relaxation subgradient methods (RSM) that are difficult to implement and that can solve non-smooth non-convex optimization problems with a high degree of elongation of level surfaces. We propose to use the transformation of the components of dual variables in the form of their square (quadratic method). We prove that the transformed dual function has a Lipschitz gradient with a quadratic method of transformation. This enables us to use efficient gradient methods to find the extremum. The above properties are confirmed by a computational experiment. With a quadratic transformation compared to a modulus transformation, it is possible to obtain a solution of the problem by relaxation subgradient methods and smooth function minimization methods (conjugate gradient method and quasi-Newtonian method) with higher accuracy and lower computational costs. The noted transformations of dual variables were used in the program module for calculating the maximum permissible emissions of enterprises (MPE) of the software package for environmental monitoring of atmospheric air (ERA-AIR).

Heterogeneous Computational Resource Allocation for NOMA: Toward Green Mobile Edge-Computing Systems

Mobile Edge Computing (MEC) is a viable solution in response to the growing demand for broadband services in the new-generation heterogeneous systems. The dense deployment of small cell networks is a key feature of next-generation radio access networks aimed at providing the necessary capacity increase. Nonetheless, the problem of green networking and service computing will be of great importance in the downlink, because the uncontrolled installation of too many small cells may increase operational costs and emit more carbon dioxide. In addition, given the resource and computational limitation of the user layer, energy efficiency (EE) and fairness assurance are critical issues in MEC-based cellular systems. Considering the user fairness criteria, this paper proposes a dynamic optimization model which maximizes the total UL/DL EE along with satisfying the necessary QoS constraints. Based on the non-convex characteristics of the EE maximization problem, the mathematical model can be divided into two separate subproblems, i.e., computational carrier scheduling and resource allocation. So that, a subgradient method is applied for the computational resource allocation and also successive convex approximation (SCA) and dual decomposition methods are adopted to solve the max-min fairness problem. The simulation results exhibit considerable EE improvement for various traffic models in addition to guaranteeing the fairness requirements. It also proved that the proposed computational partitioning scheme managed to significantly improve the total throughput for mobile computing services.

Energy-Efficient Resource Allocation for Dual-NOMA-UAV Assisted Internet of Things

Employing unmanned aerial vehicles (UAVs) characterized by low cost, high maneuverability, and on-demand deployment as aerial base stations (BSs) of Internet of Things (IoT) can guarantee communication performance in the absence of terrestrial BSs. However, the limited energy budget of UAV constrains its development. In this paper, a dual-UAV-assisted IoT using non-orthogonal multiple access (NOMA) is proposed to improve IoT capacity. To reduce energy consumption of the UAVs while ensuring a certain throughput, a joint resource optimization problem of communication scheduling, transmit power and motion parameters of UAVs is formulated to maximize energy efficiency of UAVs. To solve the proposed non-convex optimization problem, we present an alternating iterative optimization algorithm to alternately optimize three sub-problems: communication scheduling optimization, UAV transmit power optimization and UAV motion parameters optimization, each of which can be converted into convex optimization and solved using Lagrange multiplier method, subgradient descent method and successive convex approximation (SCA). The numerical results show that optimizing UAV motion parameters can effectively improve energy efficiency of UAVs, and the proposed dual-NOMA-UAV assisted IoT can achieve higher energy efficiency than the orthogonal multiple access (OMA)-UAV assisted IoT.

An Inertial Subgradient Extragradient Method for Approximating Solutions to Equilibrium Problems in Hadamard Manifolds

In this work, we are concerned with the iterative approximation of solutions to equilibrium problems in the framework of Hadamard manifolds. We introduce a subgradient extragradient type method with a self-adaptive step size. The use of a step size which is allowed to increase per iteration is to avoid the dependence of our method on the Lipschitz constant of the underlying operator as has been the case in recent articles in this direction. In general, operators satisfying weak monotonicity conditions seem to be more applicable in practice. By using inertial and viscosity techniques, we establish a convergence result for solving a pseudomonotone equilibrium problem under some appropriate conditions. As applications, we use our method to solve some theoretical optimization problems. Finally, we present some numerical illustrations in order to demonstrate the quantitative efficacy and superiority of our proposed method over a previous method present in the literature.

Dynamic Reliability Management of Multigateway IoT Edge Computing Systems

The emerging paradigm of edge computing envisions to overcome the shortcomings of cloud-centric Internet of Things (IoT) by providing data processing and storage capabilities closer to the source of data. Accordingly, IoT edge devices, with the increasing demand of computation workloads on them, are prone to failures more than ever. Hard failures in hardware due to aging and reliability degradation are particularly important since they are irrecoverable, requiring maintenance for the replacement of defective parts, at high costs. In this article, we propose a novel dynamic reliability management (DRM) technique for multigateway IoT edge computing systems to mitigate degradation and defer early hard failures. Taking advantage of the edge computing architecture, we utilize gateways for computation offloading with the primary goal of maximizing the battery lifetime of edge devices, while satisfying the Quality of Service (QoS) and reliability requirements. We present a two-level management scheme, which work together to 1) choose the offloading rates of edge devices; 2) assign edge devices to gateways; and 3) decide multihop data flow routes and rates in the network. The offloading rates are selected by a hierarchical multitimescale distributed controller. We assign edge devices by solving a bottleneck generalized assignment problem (BGAP) and compute optimal flows in a fully distributed fashion, leveraging the subgradient method. Our results, based on real measurements and trace-driven simulation, demonstrate that the proposed scheme can achieve a similar battery lifetime and better QoS compared to the state-of-the-art approaches while satisfying reliability requirements, where other approaches fail by a large margin.

Lyapunov stability of the subgradient method with constant step size

We consider the subgradient method with constant step size for minimizing locally Lipschitz semi-algebraic functions. In order to analyze the behavior of its iterates in the vicinity of a local minimum, we introduce a notion of discrete Lyapunov stability and propose necessary and sufficient conditions for stability.

Scenario decomposable subgradient projection method for two-stage stochastic programming with convex risk measures

Scenario decomposable subgradient projection method for two-stage stochastic programming with convex risk measures

S-Iteration inertial subgradient extragradient method for variational inequality and fixed point problems

In developing iterative methods for approximating solutions of optimization problems, one of the major goals of researchers is to construct efficient iterative schemes. Over the years, different techniques have been devised by authors to achieve this goal. In this paper, we study a non-Lipschitz pseudomonotone variational inequality problem with common fixed points constraint of Bregman quasi-nonexpansive mappings. We introduce a new iterative method for approximating the solution of this problem in a more general framework of reflexive Banach spaces. Our method employs several techniques to achieve high level of efficiency. One of the techniques employed is the S-iteration process, a method known to be highly efficient in comparison with several of the well-known methods. Moreover, our proposed algorithm utilizes the inertial technique and a non-monotonic self-adaptive step size to guarantee high rate of convergence and easy implementation. Unlike several of the existing results on variational inequality problem with non-Lipschitz operator, the design of our method does not involve any linesearch technique. We obtain strong convergence result for the proposed algorithm without the sequentially weakly continuity condition often assumed by authors to guarantee convergence when solving pseudomonotone variational inequality problems. Furthermore, we apply our result to study utility-based bandwidth allocation problem and image restoration problem. Finally, we present several numerical experiments to demonstrate the efficiency of our proposed method in comparison with existing state-of-the-art methods. Our result extends and improves several of the recently announced results in this direction in the literature.

Subgradient extragradient method with double inertial steps for quasi-monotone variational inequalities

In this paper, we present a modified subgradient extragradient method with double inertial extrapolation terms and a non-monotonic adaptive step size for solving quasi-monotone and Lipschitz continuous variational inequalities in real Hilbert spaces. Under some suitable conditions, we obtain the weak convergence theorem of our proposed algorithm. Moreover, strong convergence is obtained when the cost operator is strongly pseudo-monotone and Lipschitz continuous. Finally, several numerical results illustrate the effectiveness and competitiveness of our algorithm.

An inertial subgradient extragradient method with Armijo type step size for pseudomonotone variational inequalities with non-Lipschitz operators in Banach spaces

In this paper, we study the pseudomonotone variational inequality problems with non-Lipschitz operators. We propose an inertial subgradient extragradient method with Halpern technique and Armijo type step size for approximating the solution of the problem in the framework of 2-uniformly convex real Banach spaces. We prove that the sequence generated by our proposed method converges strongly to the solution of the problem under some mild conditions and without the weak sequential continuity condition often assumed by authors in solving pseudomonotone variational inequality problems. Finally, we provide some numerical experiments for the proposed method in comparison with other existing methods in the literature. Our result extends and improves several of the existing results in the current literature in this direction.