Gradient-based Iterative Methods Research Articles

Gradient-based Iterative Identification Method for Non-uniformly Sampled Input-nonlinear Systems

Gradient-based Iterative Identification Method for Non-uniformly Sampled Input-nonlinear Systems

An efficient optimization of measurement matrix for compressive sensing

Compressive sensing (CS) is a new paradigm for signal acquisition and reconstruction, which can reconstruct the signal at less than the Nyquist sampling rate. The sampling of the signal occurs through a measurement matrix (MM); thus, MM generation is significant in the context of the CS framework. In this paper, an optimization algorithm is introduced for the generation of the MM of CS based on Restricted Isometric Property (RIP) mandates that eigenvalues of the sensing matrix fall within an interval also minimizes the mutual coherence of the sensing matrix (i.e. the product of the MM and sparsifying matrix). A novel gradient-based iterative optimization method is used to reduce the eigenvalues of the sensing matrix by SVD decomposition. Meanwhile, the proposed algorithm can also reduce the operational complexity. Experimental results and analysis prove that the optimized MM reduces the maximum mutual and average mutual coherence between the MM and the sparsifying basis, which shows the effectiveness of the proposed algorithm over some state-of-art works.

Gradient-based local formulations of the Vickrey–Clarke–Groves mechanism for truthful minimization of social convex objectives

We propose a gradient-based iterative method yielding a truthfulness preserving implementation of the Vickrey–Clarke–Groves mechanism for minimization of social convex objectives. The approach is guaranteed to return, in the limit, the same efficient outcomes of the VCG method, while improving its privacy limitations and reducing its communication requirements. Its performance is investigated through an illustrative example of vehicles coordination.

Joint task offloading and resource optimization in NOMA-based vehicular edge computing: A game-theoretic DRL approach

Vehicular edge computing (VEC) becomes a promising paradigm for the development of emerging intelligent transportation systems. Nevertheless, the limited resources and massive transmission demands bring great challenges on implementing vehicular applications with stringent deadline requirements. This work presents a non-orthogonal multiple access (NOMA) based architecture in VEC, where heterogeneous edge nodes are cooperated for real-time task processing. We derive a vehicle-to-infrastructure (V2I) transmission model by considering both intra-edge and inter-edge interferences and formulate a cooperative resource optimization (CRO) problem by jointly optimizing the task offloading and resource allocation, aiming at maximizing the service ratio. Further, we decompose the CRO into two subproblems, namely, task offloading and resource allocation. In particular, the task offloading subproblem is modeled as an exact potential game (EPG), and a multi-agent distributed distributional deep deterministic policy gradient (MAD4PG) is proposed to achieve the Nash equilibrium. The resource allocation subproblem is divided into two independent convex optimization problems, and an optimal solution is proposed by using a gradient-based iterative method and KKT condition. Finally, we build the simulation model based on real-world vehicular trajectories and give a comprehensive performance evaluation, which conclusively demonstrates the superiority of the proposed solutions.

On the relaxed gradient-based iterative methods for the generalized coupled Sylvester-transpose matrix equations

In this paper, we propose the full-rank and reduced-rank relaxed gradient-based iterative algorithms for solving the generalized coupled Sylvester-transpose matrix equations. We provide analytically the necessary and sufficient condition for the convergence of the proposed iterative algorithm and give explicitly the optimal step size such that the convergence rate of the algorithm is maximized. Some numerical examples are examined to confirm the feasibility and efficiency of the proposed algorithms.

On RGI Algorithms for Solving Sylvester Tensor Equations

This paper is concerned with studying the relaxed gradient-based iterative method based on tensor format to solve the Sylvester tensor equation. From the information given by the previous steps, we further develop a modified relaxed gradient-based iterative method which converges faster than the method above. Under some suitable conditions, we prove that the introduced methods are convergent to the unique solution for any initial tensor. At last, we provide some numerical examples to show that our methods perform much better than the GI algorithm proposed by Chen and Lu (Math. Probl. Eng. 2013) both in the number of iteration steps and the elapsed CPU time.

Sparse random feature maps for the item-multiset kernel

Random feature maps are a promising tool for large-scale kernel methods. Since most random feature maps generate dense random features causing memory explosion, it is hard to apply them to very-large-scale sparse datasets. The factorization machines and related models, which use feature combinations efficiently, scale well for large-scale sparse datasets and have been used in many applications. However, their optimization problems are typically non-convex. Therefore, although they are optimized by using gradient-based iterative methods, such methods cannot find global optimum solutions in general and require a large number of iterations for convergence. In this paper, we define the item-multiset kernel, which is a generalization of the itemset kernel and dot product kernels. Unfortunately, random feature maps for the itemset kernel and dot product kernels cannot approximate the item-multiset kernel. We thus develop a method that converts an item-multiset kernel into an itemset kernel, enabling the item-multiset kernel to be approximated by using a random feature map for the itemset kernel. We propose two random feature maps for the itemset kernel, which run faster and are more memory efficient than the existing feature map for the itemset kernel. They also generate sparse random features when the original (input) feature vector is sparse and thus linear models using proposed methods . Experiments using real-world datasets demonstrated the effectiveness of the proposed methodology: linear models using the proposed random feature maps ran from 10 to 100 times faster than ones based on existing methods.

Efficient implicit Lagrangian twin parametric insensitive support vector regression via unconstrained minimization problems

In this paper, an efficient implicit Lagrangian twin parametric insensitive support vector regression is proposed which leads to a pair of unconstrained minimization problems, motivated by the works on twin parametric insensitive support vector regression (Peng: Neurocomputing. 79, 26–38, 2012), and Lagrangian twin support vector regression (Balasundaram and Tanveer: Neural Comput. Applic. 22(1), 257–267, 2013). Since its objective function is strongly convex, piece-wise quadratic and differentiable, it can be solved by gradient-based iterative methods. Notice that its objective function having non-smooth ‘plus’ function, so one can consider either generalized Hessian, or smooth approximation function to replace the ‘plus’ function and further apply the simple Newton-Armijo step size algorithm. These algorithms can be easily implemented in MATLAB and do not require any optimization toolbox. The advantage of this method is that proposed algorithms take less training time and can deal with data having heteroscedastic noise structure. To demonstrate the effectiveness of the proposed method, computational results are obtained on synthetic and real-world datasets which clearly show comparable generalization performance and improved learning speed in accordance with support vector regression, twin support vector regression, and twin parametric insensitive support vector regression.

Modified Jacobi-Gradient Iterative Method for Generalized Sylvester Matrix Equation

We propose a new iterative method for solving a generalized Sylvester matrix equation A1XA2+A3XA4=E with given square matrices A1,A2,A3,A4 and an unknown rectangular matrix X. The method aims to construct a sequence of approximated solutions converging to the exact solution, no matter the initial value is. We decompose the coefficient matrices to be the sum of its diagonal part and others. The recursive formula for the iteration is derived from the gradients of quadratic norm-error functions, together with the hierarchical identification principle. We find equivalent conditions on a convergent factor, relied on eigenvalues of the associated iteration matrix, so that the method is applicable as desired. The convergence rate and error estimation of the method are governed by the spectral norm of the related iteration matrix. Furthermore, we illustrate numerical examples of the proposed method to show its capability and efficacy, compared to recent gradient-based iterative methods.

The steepest descent of gradient-based iterative method for solving rectangular linear systems with an application to Poisson\u2019s equation

We introduce an effective iterative method for solving rectangular linear systems, based on gradients along with the steepest descent optimization. We show that the proposed method is applicable with any initial vectors as long as the coefficient matrix is of full column rank. Convergence analysis produces error estimates and the asymptotic convergence rate of the algorithm, which is governed by the term sqrt {1-kappa^{-2}}, where κ is the condition number of the coefficient matrix. Moreover, we apply the proposed method to a sparse linear system arising from a discretization of the one-dimensional Poisson equation. Numerical simulations illustrate the capability and effectiveness of the proposed method in comparison to the well-known and recent methods.

Two-stage Gradient-based Iterative Estimation Methods for Controlled Autoregressive Systems Using the Measurement Data

This paper considers the parameter identification problems of controlled autoregressive systems using observation information. According to the hierarchical identification principle, we decompose the controlled autoregressive system into two subsystems by introducing two fictitious output variables. Then a two-stage gradient-based iterative algorithm is proposed by means of the iterative technique. In order to improve the performance of the tracking the time-varying parameters, we derive a two-stage multi-innovation gradient-based iterative algorithm based on the multi-innovation identification theory. Finally, an example is provided to illustrate the effectiveness of the proposed algorithms.

Gradient-based iterative identification method for multivariate equation-error autoregressive moving average systems using the decomposition technique

This paper studies the parameter estimation problems of multivariate equation-error autoregressive moving average systems. Firstly, a gradient-based iterative algorithm is presented as a comparison. In order to improve the computational efficiency and the parameter estimation accuracy, a decomposition-based gradient iterative algorithm is presented by using the decomposition technique. The key is to transform an original system into two subsystems and to estimate the parameters of each subsystem, respectively. Compared with the gradient-based iterative algorithm, the decomposition-based algorithm requires less computational efforts, and the simulation results indicate that this algorithm is effective.

Quasi-Newton methods: superlinear convergence without line searches for self-concordant functions

ABSTRACTWe consider the use of a curvature-adaptive step size in gradient-based iterative methods, including quasi-Newton methods, for minimizing self-concordant functions, extending an approach first proposed for Newton's method by Nesterov. This step size has a simple expression that can be computed analytically; hence, line searches are not needed. We show that using this step size in the BFGS method (and quasi-Newton methods in the Broyden convex class other than the DFP method) results in superlinear convergence for strongly convex self-concordant functions. We present numerical experiments comparing gradient descent and BFGS methods using the curvature-adaptive step size to traditional methods on deterministic logistic regression problems, and to versions of stochastic gradient descent on stochastic optimization problems.

Nesterov’s accelerated gradient method for nonlinear ill-posed problems with a locally convex residual functional

In this paper, we consider Nesterov’s accelerated gradient method for solving nonlinear inverse and ill-posed problems. Known to be a fast gradient-based iterative method for solving well-posed convex optimization problems, this method also leads to promising results for ill-posed problems. Here, we provide convergence analysis of this method for ill-posed problems based on the assumption of a locally convex residual functional. Furthermore, we demonstrate the usefulness of the method on a number of numerical examples based on a nonlinear diagonal operator and on an inverse problem in auto-convolution.

Multigrid optimization for DNS-based optimal control in turbulent channel flows

The use of PDE-constrained optimization techniques in combination with transient three-dimensional turbulent flow simulations such as Direct Numerical Simulation (DNS) or Large-Eddy Simulation (LES) involves large computational cost and memory resources. To date, the minimization of a DNS-based cost functional is typically achieved by applying classical single-grid gradient-based iterative methods of quasi-Newton or non-linear conjugate gradient type. In the current study, a multigrid optimization (MG/OPT) strategy is investigated in order to speed up gradient-based algorithms designed for large scale optimization problems. The method employs a hierarchy of optimization problems defined on different representation levels. It aims to reduce the computational resources associated with the cost functional improvement on the finest level. We apply the MG/OPT method in the context of direct numerical simulations of a fully developed channel flow problem. The performance of the multigrid optimization technique is compared against the single-grid optimization method in terms of equivalent function and gradient evaluations. Also the influence of the optimization problem properties and algorithmic parameters are investigated. It is found that, in some cases, the MG/OPT method accelerates the single-grid damped L-BFGS method by a factor of four.

Regulation of soil moisture using zone model predictive control

This paper concerns the input-output model identification and zone model predictive control of an agro-hydrological system modeled by a partial differential equation. The primary control objective is to maintain the soil moisture within a desired range which is suitable for grass grow. There is also a secondary control objective which is to reduce the total irrigation amount. First, a linear parameter varying (LPV) model is identified for controller design purpose using a maximum likelihood gradient-based iterative estimation method. Then, based on the LPV model, a zone model predictive control (MPC) is designed which uses an output disturbance and state observer to reduce model-plant mismatch and an asymmetric target zone to reduce irrigation amount under weather uncertainties while maintaining the soil moisture within the target range. Simulation studies show that the LPV model is a good approximation of the original nonlinear model and effectively reduces the online computational load of the MPC, and that the proposed zone MPC can lead to significant water conservation.

Adjoint Method for Increasing the Breakdown Voltage and Reducing the On-State Resistance in Wide Band Gap Power Transistors

In this article we introduce a mathematical algorithm for the optimization of the doping distribution in power semiconductor devices in order to increase the breakdown voltage and decrease the on-state resistance of these devices. The algorithm is based on the computation of the doping sensitivity functions of the breakdown voltage and on-state resistance and uses a gradient-based iterative method to compute the optimum doping profile in metal-oxide-semiconductor field-effect transistors (MOSFET), insulted gate bipolar transistors (IGBT), p-n junctions, etc. Since the optimization is performed using gradient descents the method is well-suited for the optimization of doping concentration at all the nodes of the finite element discretization simultaneously (in which the number of nodes can be as large as 104-106). Due to the large simulation time of each individual device, traditional heuristic optimization methods such as genetic algorithms, swarm-optimization, or evolutionary algorithms are practically impossible to use when the number of optimization variables is larger than 10. Numerical results for the doping sensitivity functions of the breakdown voltage and on-state voltage are presented and discussed for a standard IGBT device.

Gradient-based iterative identification methods for multivariate pseudo-linear moving average systems using the data filtering

This paper studies the parameter identification problems of multivariate pseudo-linear moving average systems. By means of the data filtering technique, a multivariate pseudo-linear moving average system is transformed into two identification models, and a filtering-based gradient iterative algorithm is presented for estimating the parameters of these two identification models interactively. The analysis indicates that the proposed filtering-based gradient iterative algorithm can achieve a higher computational efficiency than the gradient-based iterative algorithm, and the numerical simulation results demonstrate that the proposed methods are effective.

Partially coupled gradient-based iterative identification methods for multivariable output-error moving average systems

This paper focuses on the identification problems of multivariable output-error moving average systems and develops a partially coupled gradient-based iterative algorithm to estimate the parameters of the systems. The key is combining the hierarchical identification principle and the coupling identification concept to decompose a multivariable system into m subsystems (m is the number of outputs), and then to identify the subsystems one by one. Compared with the gradient-based iterative algorithm, the partially coupled gradient-based iterative algorithm requires less computational efforts. The simulation results show that the proposed algorithms are effective.

Partially coupled gradient-based iterative identification methods for multivariable output-error moving average systems

Partially coupled gradient-based iterative identification methods for multivariable output-error moving average systems