Gradient-based Optimization Algorithm Research Articles

Dictionary-based transfer learning with Universum data

Recently, transfer learning is a popular method in machine learning, which transfers the knowledge learned from source task into target task. In practice, we can obtain the third-class examples except for the positive samples or negative samples, which are called Universum data, and Universum data can improve the performance of the classifier. In this paper, we propose a dictionarybased transfer learning with Universum data method, named U-DTL. In the proposed method, we first introduce the Universum data into the proposed model by the ∊-insensitive loss. We then embed two dictionaries for the source and target domains into a new model, and put forward the similarity constraint for dictionaries between both domains to determine the relationship among samples of source and target domains. Further, we use the gradient-based optimization and SVD algorithm to alternately optimize and update the dictionaries, and utilize Lagrangian function to iteratively optimize the proposed U-DTL model to obtain the classifier. Finally, the statistic result of Wilcoxon-test has shown that the proposed U-DTL method has the better performance than previous methods. And we have conducted extensive experiments on the benchmark datasets to evaluate the performance of the proposed U-DTL method and baselines. The results show that the proposed U-DTL method makes the better performance than previous methods.

Gradient-Based Optimization Algorithm for Solving Sylvester Matrix Equation

In this paper, we transform the problem of solving the Sylvester matrix equation into an optimization problem through the Kronecker product primarily. We utilize the adaptive accelerated proximal gradient and Newton accelerated proximal gradient methods to solve the constrained non-convex minimization problem. Their convergent properties are analyzed. Finally, we offer numerical examples to illustrate the effectiveness of the derived algorithms.

Partial Shaking Moment Balancing of Spherical Parallel Robots by a Combined Counterweight and Adjusting Kinematic Parameters Approach

Spherical parallel robots (SPR) are widely used in industries and robotic rehabilitation. Designing such systems for better balance properties is still a challenge. This paper presents a work to minimize the shaking moment for a fully force-balanced SPR by combining the counterweight (CW) and adjusting the kinematic parameters (AKP). An approximate model of the shaking moment of the SPR is proposed for computational efficiency (specifically allowing for a gradient-based optimization algorithm available in MATLAB) yet without the loss of much accuracy. The effectiveness of the proposed approach has been confirmed based on simulation, especially with the software system SPACAR due to its high reliability and easy availability. Specifically, the simulation result shows that compared with the unbalanced SPR, the shaking moment of the balanced SPR can decrease by more than 90%. It is worth mentioning that the AKP approach is an excellent example of mechatronics by combining the capability of re-planning the joint motion from the end-effector motion and adjusting the kinematic parameters to redistribute the mass of the whole robot for canceling the shaking force and shaking moment—inertia-induced force and moment to the ground. In short, the main contributions of this paper are: (1) a combined CW and AKP approach to the partial moment balancing of the SPR enhanced with a simplified mathematical model of the shaking moment of the SPR, and (2) a new design of the SPR which can be fully force balanced yet partially moment balanced. A note is taken that the simplified model is under the condition that the parameters of the link have certain geometric relations, which is a limitation of our approach.

A robust topology optimization method considering bounded field parameters with uncertainties based on the variable time step parametric level-set method

In this paper, a robust topology optimization method considering bounded field parameters with uncertainties based on the variable time step parametric level-set method is proposed. Firstly, a variable time step strategy for level set function evolution based on the gradient of the level set function that uses radial basis function interpolation to realize the separation of time and space is developed, which can achieve better design results during topology optimization. Beyond that, the dimension reduction method and dimension-wise method based on the polynomials are used for characterization and quantification of bounded field parameters with uncertainties. Finally, the sensitivity of the robust optimization model is derived based on the shape derivative principle, which provides the basis for the implementation of gradient based optimization algorithm. Three examples illustrate the effectiveness, necessity and influence of important parameters of the robust topology optimization method considering bounded field parameters with uncertainties based on the variable time step parametric level-set method from different aspects.

Gradient-Based Electromagnetic Inversion for Metasurface Design Using Circuit Models

A gradient-based optimization algorithm that is capable of directly designing a metasurface at the circuit parameter level for a desired field (amplitude and phase) pattern or a desired power (phaseless) pattern on some region of interest (ROI) external to the metasurface boundary is presented. Specifically, the inversion algorithm designs the microwave admittance profile of each subwavelength element of the metasurface when a three-layer admittance model is assumed. To this end, a forward model that maps the admittance profile of each layer of the metasurface to the desired field on the ROI is developed. Then, for the inverse design problem, a data misfit cost functional is defined and minimized over the unknown admittance profile using analytically derived gradients and step lengths. The developed inversion algorithm is then utilized to design metasurfaces capable of beam forming in the far-field zone.

Minimum feature size control in level set topology optimization via density fields

A level set topology optimization approach that uses an auxiliary density field to nucleate holes during the optimization process and achieves minimum feature size control in optimized designs is explored. The level set field determines the solid-void interface and the density field describes the distribution of a fictitious porous material using the solid isotropic material with penalization. These fields are governed by two sets of independent optimization variables which are initially coupled using a penalty for hole nucleation. The strength of the density field penalization and projection is gradually increased during the optimization process to promote a 0–1 density distribution. In addition, a second penalty regulates the evolution of the density field in the void phase. The treatment of the density field combined with the second penalty mitigate the appearance of small design features. The minimum feature size of optimized designs is controlled by the radius of the linear filter applied to the density optimization variables. The structural response is predicted by the extended finite element method, the sensitivities by the adjoint method, and the optimization variables are updated by a gradient-based optimization algorithm. Numerical examples investigate the robustness of this approach with respect to algorithmic parameters and mesh refinement. The results show the applicability of the combined density level set topology optimization approach for both optimal hole nucleation and for minimum feature size control in 2D and 3D. This comes, however, at the cost of a more complex problem formulation and additional computational cost due to an increased number of optimization variables.

Efficient method for accelerating line searches in adjoint optimization of photonic devices by combining Schur complement domain decomposition and Born series expansions.

A line search in a gradient-based optimization algorithm solves the problem of determining the optimal learning rate for a given gradient or search direction in a single iteration. For most problems, this is determined by evaluating different candidate learning rates to find the optimum, which can be expensive. Recent work has provided an efficient way to perform a line search with the use of the Shanks transformation of a Born series derived from the Lippman-Schwinger formalism. In this paper we show that the cost for performing such a line search can be further reduced with the use of the method of the Schur complement domain decomposition, which can lead to a 10-fold total speed-up resulting from the reduced number of iterations to convergence and reduced wall-clock time per iteration.

Robust design optimization by spline dimensional decomposition

This article highlights new spline-empowered computational methods for solving robust design optimization (RDO) problems in complex mechanical systems. The methods are predicated on a spline dimensional decomposition (SDD) of a high-dimensional, discontinuous, or nonsmooth stochastic response for statistical moment analysis, a novel fusion of SDD and score functions for calculating the second-moment sensitivities with respect to the design variables, and standard gradient-based optimization algorithms. New closed-form formulae are derived for the design sensitivities that are concurrently determined along with the moments. The methods depend on how the statistical moment and sensitivity analyses are engaged with an optimization algorithm, engendering direct and multi-point single-step design processes. Numerical results reveal that the proposed methods deliver accurate and computationally efficient optimal solutions to RDO problems, including an industrial-scale shape design of a robotic gripper jaw.

Multi-objective optimization of automobile body frame considering weight, rigidity, and frequency for conceptual design

At the conceptual stage of product design, simplified automobile body frame constituted by thin-walled beams can effectively be used to predict global performances, including weight, rigidity, and frequency. These performances can be improved by optimizing their cross-sectional shapes (CSS) of thin-walled beams. However, it is difficult to optimize the CSS while satisfying multiple performances, because this is a multiple objectives and design variables optimization problem. The gradient-based optimization algorithms are difficult to obtain the global optimal solutions for the automobile structures. Therefore, this paper proposes an innovative multi-objective optimization method to design the CSS of automobile body by using the non-dominated sorting genetic algorithm (NSGA-II) combining with the artificial neural network. Firstly, the mechanical properties of the CSS are summarized, including open-cell, single-cell, and double-cells. These mechanical properties determine the performances of the automobile structure. Then, the multi-objective optimization model is created by using the NSGA-II while considering the weight, stiffness, and frequency, which is implemented in the self-developed CarFrame software. Finally, the proposed method is verified by optimizing the CSS for the A-pillar of automobile frame.

On the Reformulation of the Classical Stefan Problem as a Shape Optimization Problem

This article is concerned with the multi-dimensional one-phase Stefan problem, which belongs to the class of moving boundary problems. We suggest to reformulate the classical Stefan problem as a shape optimization problem, consisting of an objective functional for the moving boundary and a partial differential equation corresponding to a heat type equation. Minimizing the objective functional subject to the differential equation under consideration is equivalent to solving the Stefan problem. In order to apply gradient-based optimization algorithms, we analytically compute the shape gradient of the objective functional. A numerical example justifies our approach.

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality.

Recent theoretical studies proved that deep neural network (DNN) estimators obtained by minimizing empirical risk with a certain sparsity constraint can attain optimal convergence rates for regression and classification problems. However, the sparsity constraint requires knowing certain properties of the true model, which are not available in practice. Moreover, computation is difficult due to the discrete nature of the sparsity constraint. In this letter, we propose a novel penalized estimation method for sparse DNNs that resolves the problems existing in the sparsity constraint. We establish an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator and derive convergence rates for several learning tasks. In particular, we prove that the sparse-penalized estimator can adaptively attain minimax convergence rates for various nonparametric regression problems. For computation, we develop an efficient gradient-based optimization algorithm that guarantees the monotonic reduction of the objective function.

Isogeometric-analysis-based stiffness spreading method for truss layout optimization

A novel isogeometric analysis-based stiffness spreading method (IGA-based SSM) is proposed for the truss layout optimization design within a continuum, where the topology, geometry, and cross-sectional areas of the truss elements can be simultaneously optimized. The IGA-based SSM based on energy conservation is employed to obtain the spreading stiffness matrices of the truss elements that are embedded in weak IGA background grids. The IGA background grids are constructed using high-order continuous isogeometric elements that can overcome the limitation of discontinuous sensitivity derived by traditional C0 elements. Because the sensitivity required for the optimization can be obtained analytically, gradient-based optimization algorithms can be easily implemented. To verify the effectiveness of the IGA-based SSM for truss layout optimization design, typical illustrative examples are used, considering the compliance minimization optimization problem. The proposed method can provide a continuous and smooth sensitivity field and overcome the numerical inaccuracy caused by interpolation methods, such as the radial basis function. The different background mesh sizes, initial layouts, and orders p of nonuniform rational B-splines basis functions are investigated in truss layout optimization. The results indicate that the selection of the appropriate background mesh size can avoid the problem of discontinuous truss connections, C1-continuous isogeometric elements can provide better optimization designs with a lower computational cost, and the proposed method can provide a clear layout for different initial layouts. Moreover, the spatial truss layout optimization problem and the min–max displacement optimization problem are also studied to further verify the effectiveness of the proposed method. Finally, the proposed method is extended to solve the concentrated force diffusion problem that widely exists in the interstage adapters of launch vehicles by adding the variance constraint of reaction forces. The results indicate that the proposed method can obtain required concentrated force diffusion results with reasonable stiffness distributions.

GGA-MLP: A Greedy Genetic Algorithm to Optimize Weights and Biases in Multilayer Perceptron.

The task of designing an Artificial Neural Network (ANN) can be thought of as an optimization problem that involves many parameters whose optimal value needs to be computed in order to improve the classification accuracy of an ANN. Two of the major parameters that need to be determined during the design of an ANN are weights and biases. Various gradient-based optimization algorithms have been proposed by researchers in the past to generate an optimal set of weights and biases. However, due to the tendency of gradient-based algorithms to get trapped in local minima, researchers have started exploring metaheuristic algorithms as an alternative to the conventional techniques. In this paper, we propose the GGA-MLP (Greedy Genetic Algorithm-Multilayer Perceptron) approach, a learning algorithm, to generate an optimal set of weights and biases in multilayer perceptron (MLP) using a greedy genetic algorithm. The proposed approach increases the performance of the traditional genetic algorithm (GA) by using a greedy approach to generate the initial population as well as to perform crossover and mutation. To evaluate the performance of GGA-MLP in classifying nonlinear input patterns, we perform experiments on datasets of varying complexities taken from the University of California, Irvine (UCI) repository. The experimental results of GGA-MLP are compared with the existing state-of-the-art techniques in terms of classification accuracy. The results show that the performance of GGA-MLP is better than or comparable to the existing state-of-the-art techniques.

Gradient descent algorithm to optimize the offshore scale squeeze treatments

Scale deposition is one of the serious oilfield chemical issues which may lead to a range of downhole and production problems, including the reduction of well productivity index. Scale Inhibitor (SI) squeeze treatments are one of the most common techniques which are applied to prevent downhole scaling in production wells. A treatment typically consists of four stages, (i) a preflush, to condition the rock surface; (ii) a main treatment, where a batch of high concentration inhibitor is bullheaded into the formation; (iii) an overflush, to displace the scale inhibitor slug deeper into the near-well formation, and (iv) a shut-in stage to allow further inhibitor retention before putting the well back on production. During this backflow period, scale inhibitor is released from the rock surface into the produced water, and the scale deposition is prevented if the inhibitor concentration is above some specified Minimum Inhibitor Concentration (MIC). Due to logistic constraints in offshore wells, it is often required that the scale protection afforded by a squeeze treatment should last for some fixed design lifetime until the next treatment becomes available. For example, in the North Sea sector, well operations are often based on an annual treatment design. This paper presents a methodology to optimize the squeeze treatment design for a fixed target lifetime and this is applied for two offshore well squeeze treatments. This approach allows us to account for the operational constraints in the squeeze optimization process to treat a fixed volume of produced water, while minimizing the inhibitor neat volume and the total pumping time. A sensitivity study was performed on the inhibitor concentration, where the results showed that deploying a smaller inhibitor slug but with higher concentration is more effective than a larger slug with lower concentration, assuming a fixed volume of inhibitor and injected water. Therefore, it is recommended that the inhibitor is deployed at the maximum possible concentration, while avoiding the potential for any formation damage. Considering the same inhibitor slug (volume and concentration), this study suggests that the squeeze lifetime continuously increases with the overflush volume. This implies that the lifetime function is differentiable with respect to the overflush, and thus a gradient-based optimization algorithm, specifically the Gradient Descent (GD) may be applied to find the exact overflush volume resulting in the target lifetime. Employing this procedure for a wide range of main treatment volumes allows us to calculate the squeeze “Iso-Lifetime” curve which represents all the possible squeeze designs for the target lifetime. Thereafter, from the iso-lifetime designs, the CPB value (total cost of squeeze per treated barrel of water produced) can be minimized to find the most optimum squeeze design. This approach is shown to result in the optimum scale inhibitor squeeze treatment strategy in the long-term.

Recent progress on equation-oriented optimization of complex chemical processes

Process optimization in equation-oriented (EO) modeling environments favors the gradient-based optimization algorithms by their abilities to provide accurate Jacobian matrices via automatic or symbolic differentiation. However, computational inefficiencies including that in initial-point-finding for Newton type methods have significantly limited its application. Recently, progress has been made in using a pseudo-transient (PT) modeling method to address these difficulties, providing a fresh way forward in EO-based optimization. Nevertheless, research in this area remains open, and challenges need to be addressed. Therefore, understanding the state-of-the-art research on the PT method, its principle, and the strategies in composing effective methodologies using the PT modeling method is necessary for further developing EO-based methods for process optimization. For this purpose, the basic concepts for the PT modeling and the optimization framework based on the PT model are reviewed in this paper. Several typical applications, e.g., complex distillation processes, cryogenic processes, and optimizations under uncertainty, are presented as well. Finally, we identify several main challenges and give prospects for the development of the PT based optimization methods.

Scattering Model Guided Adversarial Examples for SAR Target Recognition: Attack and Defense

Deep Neural Networks (DNNs) based Synthetic Aperture Radar (SAR) Automatic Target Recognition (ATR) systems have shown to be highly vulnerable to adversarial perturbations that are deliberately designed yet almost imperceptible but can bias DNN inference when added to targeted objects. This leads to serious safety concerns when applying DNNs to high-stakes SAR ATR applications. Therefore, enhancing the adversarial robustness of DNNs is essential for applying DNNs to modern real-world SAR ATR systems. Toward building more robust DNN-based SAR ATR models, this article explores the domain knowledge of SAR imaging process and proposes a novel Scattering Model Guided Adversarial Attack (SMGAA) algorithm which can generate adversarial perturbations in the form of electromagnetic scattering response (called adversarial scatterers). The proposed SMGAA consists of two parts: 1) a parametric scattering model and corresponding imaging method and 2) a customized gradient-based optimization algorithm. First, we introduce the effective Attributed Scattering Center Model (ASCM) and a general imaging method to describe the scattering behavior of typical geometric structures in the SAR imaging process. By further devising several strategies to take the domain knowledge of SAR target images into account and relax the greedy search procedure, the proposed method does not need to be prudentially finetuned, and can efficiently find the effective ASCM parameters to fool the SAR classifiers and facilitate the robust model training. Comprehensive evaluations on the MSTAR dataset show that the adversarial scatterers generated by SMGAA are more robust to perturbations and transformations in the SAR processing chain than the currently studied attacks, and are effective to construct a defensive model against the malicious scatterers.

Stiffener layout optimization framework by isogeometric analysis-based stiffness spreading method

An innovative stiffener layout optimization framework is proposed based on the isogeometric analysis-based stiffness spreading method (IGA-based SSM), where the shape and size of stiffeners can be simultaneously optimized. The IGA-based SSM is employed to obtain the stiffness spreading matrices of the stiffeners that are embedded in the flat plate. The flat plate is modeled by high-order continuous isogeometric degenerated shell elements that can provide a smooth sensitivity field. Since the sensitivity required for the optimization can be obtained analytically, a gradient-based optimization algorithm can be employed. Moreover, the ground structure is used to provide the initial layouts of stiffeners and ensure the connectivity of stiffeners. A local geometry control strategy is also adopted to avoid the intersection of stiffeners in the optimization process. To verify the effectiveness and efficiency of the proposed method, typical illustrative examples are discussed in detail, considering the compliance minimization optimization problem. Compared with the continuum topology optimization method, the proposed method can provide a clear stiffener layout with a lower computational cost due to fewer design variables and fewer degrees of freedom of the optimization problem. Besides, the maximum thicknesses of stiffeners can be explicitly controlled, and it can easily solve the optimization problem of various volume fraction constraints. Finally, the proposed method is extended to solve a layout optimization problem of the engine nozzle adjusting plate under pressure loads, which obtains a good stiffener layout under the maximum displacement constraint.

Sensitivity estimation of conditional value at risk using randomized quasi-Monte Carlo

Conditional value at risk (CVaR) is a popular measure for quantifying portfolio risk. Sensitivity analysis of CVaR is common in risk management and gradient-based optimization algorithms. In this paper, we study the infinitesimal perturbation analysis estimator for CVaR sensitivity using randomized quasi-Monte Carlo (RQMC) simulation. RQMC has proved valuable in financial option pricing with a better rate of convergence compared to Monte Carlo sampling, but theoretical guarantees for this new application of RQMC shall be studied. To this end, we first prove that the RQMC-based estimator is strongly consistent under very mild conditions. Under some technical conditions, RQMC yields a mean error rate of O(n−1/2−1/(4d−2)+ϵ) for arbitrarily small ϵ>0, where d represents the dimension of RQMC points and n is the sample size. Some typical applications of CVaR sensitivity estimation are conducted to both show how the theoretical results can be applied, as well as to provide numerical results documenting the superiority of the RQMC estimator.

Chance Constrained Programs with Gaussian Mixture Models

In this paper, we discuss input modeling and solution techniques for several classes of chance constrained programs (CCPs). We propose to use Gaussian mixture models (GMM) to fit the data available and to model the randomness. We demonstrate the merits of using GMM. We consider several scenarios that arise from practical applications and analyze how the problem structures could embrace alternative optimization techniques. More specifically, for several scenarios, we study how to assess the gradient of the chance constraint and incorporate the results into gradient-based nonlinear optimization algorithms, and for a class of CCPs, we propose a spatial branch-and-bound (BB) procedure and solve the problems to global optimality. We also conduct numerical experiments to test the efficiency of our approach and propose an example of hedge fund portfolio to illustrate the practical application of the method.

Optimal state-delay control in nonlinear dynamic systems

This paper considers a class of nonlinear systems in which the control function is a time-varying state-delay. The optimal control problem is to optimize the time-varying delay and a set of time-invariant system parameters subject to lower and upper bounds. To solve this problem, we first parameterize the delay in terms of piecewise-quadratic basis functions, thus yielding a finite-dimensional approximate problem with continuous-time inequality constraints induced by the delay bounds. We then exploit the quadratic structure of the delay to convert these continuous-time constraints into a finite set of canonical point constraints. We also develop an efficient numerical method for computing the gradients of the system cost function. This method, which involves integrating an auxiliary impulsive system with time-varying advance backwards in time, can be combined with any existing gradient-based optimization algorithm to generate approximate solutions for the optimal control problem.