Non-gradient Optimization Research Articles

Fast gradient-free activation maximization for neurons in spiking neural networks

Elements of neural networks, both biological and artificial, can be described by their selectivity for specific cognitive features. Understanding these features is important for understanding the inner workings of neural networks. For a living system, such as a neuron, whose response to a stimulus is unknown and not differentiable, the only way to reveal these features is through a feedback loop that exposes it to a large set of different stimuli. The properties of these stimuli should be varied iteratively in order to maximize the neuronal response. To utilize this feedback loop for a biological neural network, it is important to run it quickly and efficiently in order to reach the stimuli that maximizes certain neurons’ activation with the least number of iterations possible. Here we present a framework with an efficient design for such a loop. We successfully tested it on an artificial spiking neural network (SNN), which is a model that simulates the asynchronous spiking activity of neurons in living brains. Our optimization method for activation maximization is based on the low-rank Tensor Train decomposition of the discrete activation function. The optimization space is the latent parameter space of images generated by SN-GAN or VQ-VAE generative models. To our knowledge, this is the first time that effective AM has been applied to SNNs. We track changes in the optimal stimuli for artificial neurons during training and show that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network. This formation of refined optimal stimuli is associated with an increase in classification accuracy. Some neurons, especially in the deeper layers, may gradually change the concepts they are selective for during learning, potentially explaining their importance for model performance. The source code of our framework, MANGO (for Maximization of neuronal Activation via Non-Gradient Optimization) is available on GitHub (https://github.com/iabs-neuro/mango)

Dynamic Machine Learning Global Optimization Algorithm and Its Application to Aerodynamics

A dynamic extreme gradient boosting (XGBoost) and MaxLIPO trust region parallel global optimization algorithm is proposed in this paper, and it is applied to the turbomachinery blade aerodynamic optimization coupled with an in-house graphics processing unit (GPU) heterogeneous accelerated compressible flow solver, AeroWhale. The algorithm combines an accurate machine learning regression model, an efficient nongradient optimization method with no hyperparameters, a dynamic update regression strategy, and double convergence criteria to achieve high optimization accuracy and efficiency. The optimization results on the test function indicate that the number of objective function calls is less than 2% of that required by a traditional genetic algorithm, which greatly reduces the optimization time. The dynamic XGBoost model ensures that the regression model accuracy near the optimum is relatively high, which is attributed to the update strategy. The error between the optimal value identified by the proposed algorithm and the theoretical value is only 0.52% after several objective function calls. Finally, the aerodynamic optimization algorithm is applied to the LS89 high-pressure turbine, and the total pressure loss is reduced by 13.16%. The sensitivity of each optimization feature to the objective function is determined, showing that the blade suction surface control point near the trailing edge has the greatest impact on aerodynamic performance.

Design optimization of metamaterials to control waves in cylindrical rods

Metamaterials can control different modes of waves by tuning their bandgap structures. Various design strategies are implemented to obtain desired responses of metamaterials by solving multiple forward problems using parametric sweeps, topology, and shape optimization problems. This study focuses on the design of cylindrical metamaterials to control wave propagation in a cylindrical rod using a shape optimization approach. Topology optimization requires relatively more computational resources and time as well as it also gives relatively complex structures in comparison with shape optimization. Optimal widths of layered elastic materials arranged periodically along an axis, on subsurface, and an outer surface of cylindrical surface are obtained by solving multiple time-dependent shape optimization problems. Optimization problems are solved using the finite element method and non-gradient optimization algorithm. Single and multi-objective functions are defined to reduce only axial and both axial as well as radial displacements integrated over the end surface, and monochromatic Gaussian input pulse is considered in this study. Every design solution obtained during each design iteration can be easily manufactured as a weak constraint is applied to the total length of the metamaterial. Prior knowledge of possible bandgaps helps to set an effective optimization problem, but it is not a necessary condition.

Pushing the Limits of Functionality-Multiplexing Capability in Metasurface Design Based on Statistical Machine Learning.

As 2D metamaterials, metasurfaces provide an unprecedented means to manipulate light with the ability to multiplex different functionalities in a single planar device. Currently, most pursuits of multifunctional metasurfaces resort to empirically accommodating more functionalities at the cost of increasing structural complexity, with little effort to investigate the intrinsic restrictions of given meta-atoms and thus the ultimate limits in the design. In this work, it is proposed to embed machine-learning models in both gradient-based and nongradient optimization loops for the automatic implementation of multifunctional metasurfaces. Fundamentally different from the traditional two-step approach that separates phase retrieval and meta-atom structural design, the proposed end-to-end framework facilitates full exploitation of the prescribed design space and pushes the multifunctional design capacity to its physical limit. With a single-layer structure that can be readily fabricated, metasurface focusing lenses and holograms are experimentally demonstrated in the near-infrared region. They show up to eight controllable responses subjected to different combinations of working frequencies and linear polarization states, which are unachievable by the conventional physics-guided approaches. These results manifest the superior capability of the data-driven scheme for photonic design, and will accelerate the development of complex devices and systems for optical display, communication, and computing.

Manifold Neural Network With Non-Gradient Optimization.

Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent and thus has a slow convergence. In addition, softmax, as a decision layer, may ignore the distribution information of the data during classification. Aiming to tackle the referred problems, we propose a novel manifold neural network based on non-gradient optimization, i.e., the analytical-form solutions. Considering that the activation function is generally invertible, we reconstruct the network via forward ridge regression and low-rank backward approximation, which achieve rapid convergence. Moreover, by unifying the flexible Stiefel manifold and adaptive support vector machine, we devise the novel decision layer which efficiently fits the manifold structure of the data and label information. Consequently, a jointly non-gradient optimization method is designed to generate the network with analytical-form results. Furthermore, an acceleration strategy is utilize to reduce the time complexity for handling high dimensional datasets. Eventually, extensive experiments validate the superior performance of the model.

Optimization-based model calibration of marginal and joint output distributions utilizing analytical gradients

Optimization-based model calibration (OBMC) determines statistical parameters of unknown input variables by optimizing statistical similarity between simulations and observations. Gradient-based optimization algorithms have been utilized for OBMC since they require less computational cost compared with non-gradient optimization algorithms. However, OBMC using numerical gradients such as finite differences has disadvantages in the aspect of computational cost and numerical instability. For these reasons, a previous study derived analytical gradients under the assumption that performance functions are a linear single response and unknown inputs follow a normal distribution. However, to apply the model calibration to general engineering problems, analytical gradients need to be derived without any assumption. Therefore, it is proposed in this study to derive analytical gradients of marginal and joint likelihood functions to calibrate both linear and nonlinear responses. In addition, the analytical gradients are derived for five distribution types of unknown input random variables to be estimated: normal, lognormal, Gumbel, extreme, and uniform distributions. For derivation of the analytical gradients, the chain rule is utilized to combine the derivatives of the inverse function of cumulative distribution function (CDF), performance function, and joint likelihood function. Numerical examples and one engineering example using finite element analysis (FEA) are employed to verify the accuracy and efficiency of the proposed analytical gradient, and it is demonstrated that accurately calibrated output responses can be obtained by the proposed method by finding a better optimum in OBMC.

Realization of full and directional band gap design by non-gradient topology optimization in acoustic metamaterials

Full and directional band gap acoustic metamaterials offer the promising ability of controlling and prohibiting propagation of acoustic/elastic waves in specified frequency ranges or directions. However, there is no guarantee that the optimization problem will be solved successfully with the existing gradient-based topology optimization methods because of the complexity of the problem and strong dependence on initial guesses. In this study, we developed a systematic topological optimization method based on material-field series expansion (MFSE) framework for full and directional band gap acoustic metamaterials design. Herein, the number of involved design variables of the MFSE method is greatly reduced to no more than 50, and an optimal band gap design of acoustic metamaterials is therefore able to be obtained with non-gradient optimization algorithms. Then, the Kriging-based optimization algorithm with a self-adaptive strategy is adopted for solving the optimization problem. The optimized designs finally converge to the orderly material distribution and numerical validations show improved full and directional propagation properties as expected. The realization of two-dimension acoustic metamaterials demonstrates that the proposed method can generate meaningful optimized topologies of PnCs for the full frequency band and directional propagation with a broad frequency range.

Adjustments for Wind-Induced Undercatch in Snowfall Measurements Based on Precipitation Intensity

AbstractAdjustments for the wind-induced undercatch of snowfall measurements use transfer functions to account for the expected reduction of the collection efficiency with increasing the wind speed for a particular catching-type gauge. Based on field experiments or numerical simulation, collection efficiency curves as a function of wind speed also involve further explanatory variables such as surface air temperature and/or precipitation type. However, while the wind speed or wind speed and temperature approach is generally effective at reducing the measurement bias, it does not significantly reduce the root-mean-square error (RMSE) of the residuals, implying that part of the variance is still unexplained. In this study, we show that using precipitation intensity as the explanatory variable significantly reduces the scatter of the residuals. This is achieved by optimized curve fitting of field measurements from the Marshall Field Site (Colorado, United States), using a nongradient optimization algorithm to ensure optimal binning of experimental data. The analysis of a recent quality-controlled dataset from the Solid Precipitation Intercomparison Experiment (SPICE) campaign of the World Meteorological Organization confirms the scatter reduction, showing that this approach is suitable to a variety of locations and catching-type gauges. Using computational fluid dynamics simulations, we demonstrate that the physical basis of the reduction in RMSE is the correlation of precipitation intensity with the particle size distribution. Overall, these findings could be relevant in operational conditions since the proposed adjustment of precipitation measurements only requires wind sensor and precipitation gauge data.

Quantum Bacterial Foraging Optimization: From Theory to MIMO System Designs

This article develops a quantum bacterial foraging optimization (QBFO) algorithm, a quantum intelligence algorithm based on quantum computing and bacterial foraging optimization (BFO), with application in MIMO system optimization designs. In QBFO, a multiqubit is used to represent a bacterium, and a quantum rotation gate is used to mimic chemotaxis. Because the quantum bacterium with multiqubit has the advantage that it can represent a linear superposition of states (binary solutions) in search space probabilistically, the proposed QBFO algorithms shows better performance on solving combinatorial optimization problems than its classical counterpart BFO and Quantum Genetic Algorithm (QGA), especially for parallel non-gradient optimization. A sparse channel estimation scheme based on QBFO with adaptive phase rotation (AQBFO) in 3D MIMO system is proposed, and simulation results show that AQBFO achieved a better performance than existing algorithms including least squares (LS), iteratively reweighted least squares (IRLS), matching pursuit (MP), and orthogonal matching pursuit (OMP). We further improve some critical aspects such as reproduction and dispersal processes of AQBFO, propose an improved IQBFO algorithm, and apply it for interference coordination in 3D multi-cell multi-user MIMO systems, aiming to maximize the spectral efficiency. It considers user fairness and jointly optimizes cell-center and cell-edge user specific antenna downtilts and power to maximize each user's sum rate. This problem is a combinatorial non-convex optimization problem that cannot be solved by the traditional Karush-Kuhn-Tucker Lagrangian algorithm whereas the IQBFO algorithm solves it effectively.

Particle Swarm Optimization to Obtain Weights in Neural Network

This research discusses the use of a class of heuristic optimization to obtain the weights in neural network model for time series prediction. In this case, Feed Forward Neural Network (FFNN) was chosen as the class of network architecture. The heuristic algorithm determined to obtain the weights in network was Particle Swarm Optimization (PSO). It is a non-gradient optimization technique. This method was used for optimizing the connection weights of network. The lags used as the input were selected based on the strong relationship with the current. The eight architectures were conducted to improve the accuracy of the neural network model. In each architecture, we repeated the running thirty times to get the statistics of mean and variance. The comparison of the performance of various architectures based on the minimum MSE and the stability of the results is presented in this paper. The optimal number of neurons in hidden layer was determined by these criteria. The proposed procedure was applied in air pollution data, i.e. Solid Particulate Matter (SPM). The results showed that the proposed procedure gave promising results in terms of prediction accuracy. A few neurons in hidden layer are strongly recommended in choosing the optimal architecture.

Comparison of gradient and nongradient algorithms in the structural optimization course

Gradient and nongradient optimization algorithms are currently available for structural design in structural optimization course. Despite the successful application of gradient algorithm in structural optimization, nongradient algorithm is also extensively adopted to solve the structural optimization problem. However, the efficiency of nongradient algorithm has caused a heated debate recently. To clarify this issue for the graduate students, sequential linear programming and genetic algorithm are, respectively, chosen as the representatives of gradient and nongradient algorithms to solve truss size optimization problem. Firstly, the size optimization formulations of truss structure for sequential linear programming and genetic algorithm are summarized, respectively. Secondly, an educational finite element software for truss structure is developed by using the object-oriented programming to create the software framework. This study aims to provide an open-source, extensible, and benchmarking software, which do assist the students to understand the structural optimization process in engineering education. Finally, two benchmarking examples are introduced to compare the efficiency and accuracy of sequential linear programming and genetic algorithm.

A terahertz spatial power combiner based on 2D periodic hole-shaped grating using nongradient optimization method

ABSTRACTThis paper describes a terahertz spatial power combiner based on a periodic hole-shaped hologram for a two-dimensional (2D) horn antenna array. The combiner consists of a 2D array of rectangular horns, a dielectric hole-shaped hologram, a focusing reflector, and a receiving horn antenna. The hologram is realized as 2D periodic holes, which transforms the beams of the antenna array into a pseudoplane wave. The transformed wave is coupled into a rectangular output horn antenna by a parabolic reflector with a shaped surface relief. The field distribution on the receiving plane can be obtained by using a nongradient optimization method to optimize the reflector surface. An example operating at 300 GHz is described. The theoretical power combining efficiency is about 63.1%. The computed field distribution agrees well with the result obtained using the commercial software FEKO Suite.

Estimating the base material properties of sintered metallic hollow spheres by inverse engineering procedure

The base material properties of single sintered metallic hollow spheres cannot be determined directly because of their small size, with the outer diameter ranging from 1 to 15mm. The paper proposes an inverse engineering procedure for their determination by applying the computational optimisation based on an extensive experimental testing data. Initially, the compressive response of single iron hollow spheres of selected sizes of 2 and 3mm in diameter has been determined by quasi-static and dynamic experimental testing. The exact geometrical data of the analyzed spheres was obtained by μCT scanning and then used for creation of the corresponding FE models. These were subjected to the same loading conditions in computational simulations as those used in experimental testing. The non-gradient optimization process based on genetic algorithm was applied to define appropriate base material properties of analyzed spheres, which returned the same response of computational models as the one recorded in experiments. This way the base material properties of sintered iron hollow spheres were obtained for the first time, which can be used for computational studies of the metallic hollow sphere structures behavior.

Task-based design optimization of serial robot manipulators

In this article a direct search non-gradient design optimization method is proposed to determine the optimal task time and optimal parameters of serial robot manipulators. The design variables are treated as continuous variables associated with the motion trajectory profile and robot's link lengths and masses. The method adopted in this study can be applied to any serial redundant or non-redundant manipulator that has rigid links and known kinematic and dynamic models with free motions or motions along specified paths. Trajectory can be planned in Cartesian or joint space under kinodynamic constraints. The cost function to be minimized is weighted terms of task time and joint torques quadratic average. Examples on 3-R planar and spatial serial robot manipulators are given to illustrate the method where the kinematic and dynamic models and the optimization method are developed in MATLAB®.

A survey of non-gradient optimization methods in structural engineering

In this paper, we present a review on non-gradient optimization methods with applications to structural engineering. Due to their versatility, there is a large use of heuristic methods of optimization in structural engineering. However, heuristic methods do not guarantee convergence to (locally) optimal solutions. As such, recently, there has been an increasing use of derivative-free optimization techniques that guarantee optimality. For each method, we provide a pseudo code and list of references with structural engineering applications. Strengths and limitations of each technique are discussed. We conclude with some remarks on the value of using methods customized for a desired application.

Suspension system performance optimization with discrete design variables

Suspension systems on commercial vehicles have become an important feature meeting the requirements from costumers and legislation. The performance of the suspension system is often limited by available catalogue components. Additionally the suspension performance is restricted by the travel speed which highly influences the ride comfort. In this article a suspension system for an articulated dump truck is optimized in sense of reducing elapsed time for two specified duty cycles without violating a certain comfort threshold level. The comfort threshold level is here defined as a whole-body vibration level calculated by ISO 2631-1. A three-dimensional multibody dynamics simulation model is applied to evaluate the suspension performance. A non-gradient optimization routine is used to find the best possible combination of continuous and discrete design variables including the optimum operational speed without violating a set of side constraints. The result shows that the comfort level converges to the comfort threshold level. Thus it is shown that the operational speed and hence the operator input influences the ride comfort level. Three catalogue components are identified by the optimization routine together with a set of continuous design variables and two operational speeds one for each load case. Thus the work demonstrates handling of human factors in optimization of a mechanical system with discrete and continuous design variables.

Crashworthiness optimization of foam-filled tapered thin-walled structure using multiple surrogate models

Despite the rapid growth of computing power and continuing advancements in numerical techniques, significant complexity exists when applying traditional sensitivity based optimization to such highly nonlinear problems as crashworthiness design. As a major alternative, surrogate modeling techniques have proven considerably effective. However the challenge remains how to determine the most suitable surrogate scheme for modeling nonlinear responses and conducting optimization. This paper presents a comparative study on the different surrogate models, such as polynomial response surface (PRS), Kriging (KRG), support vector regression (SVR) and radial basis function (RBF), which have been widely used for a variety of engineering problems, thereby gaining insights into their relative performance and features in computational modeling and design. In this study, a foam-filled tapered thin-walled structure is exemplified. Both the gradient and non-gradient algorithms, specifically sequential quadratic programming (SQP) and particle swarm optimization (PSO), are used for these abovementioned four surrogate models, respectively. The design results demonstrate that simultaneous use of different surrogate models can be essential for both gradient and non-gradient optimization algorithms because they may generate different outcomes in the crashworthiness design.

Retraction of “A Biologically Inspired Intelligent PID Controller Tuning for AVR Systems”

This paper proposes a hybrid approach involving Genetic Algorithm (GA) and Bacterial Foraging (BF) for tuning the PID controller of an AVR. Recently the social foraging behavior of E. coli bacteria has been used to solve optimization problems. We first illustrate the proposed method using four test functions and the performance of the algorithm is studied with an emphasis on mutation, crossover, variation of step sizes, chemotactic steps, and the life time of the bacteria. Further, the proposed algorithm is used for tuning the PID controller of an AVR. Simulation results are very encouraging and this approach provides us a novel hybrid model based on foraging behavior with a possible new connection between evolutionary forces in social foraging and distributed non-gradient optimization algorithm design for global optimization over noisy surfaces.

Design of compliant mechanism using hyper radial basis function network and reanalysis formulation

The design of compliant mechanisms often involves multiple minima. Since non-gradient type global optimization techniques cannot handle as many design variables as the commonly used, gradient based SIMP method, a hyper radial basis function network is used in this paper for Particle Swarm Optimization. An exact reanalysis formulation is presented which computes the strain energy from the solution of the mutual energy problem. The design of motion inverter is presented to demonstrate the proposed approach. The use of hyper radial basis function network results in checkerboard-free, smooth topology using very crude finite element analysis mesh, whereas the reanalysis formulation cuts down the analysis time into half. Thus the design process is made computationally efficient and less dependent on human interpretation of the optimal topology.

Bacterial Foraging Optimization

The bacterial foraging optimization (BFO) algorithm mimics how bacteria forage over a landscape of nutrients to perform parallel nongradient optimization. In this article, the author provides a tutorial on BFO, including an overview of the biology of bacterial foraging and the pseudo-code that models this process. The algorithms features are briefly compared to those in genetic algorithms, other bio-inspired methods, and nongradient optimization. The applications and future directions of BFO are also presented.