Neural Network Optimization Method Research Articles

Lifted Proximal Operator Machines

We propose a new optimization method for training feedforward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feedforward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machine (LPOM). LPOM is block multiconvex in all layer-wise weights and activations. This allows us to use block coordinate descent to update the layer-wise weights and activations. Most notably, we only use the mapping of the activation function itself, rather than its derivative, thus avoiding the gradient vanishing or blow-up issues in gradient based training methods. So our method is applicable to various non-decreasing Lipschitz continuous activation functions, which can be saturating and non-differentiable. LPOM does not require more auxiliary variables than the layer-wise activations, thus using roughly the same amount of memory as stochastic gradient descent (SGD) does. Its parameter tuning is also much simpler. We further prove the convergence of updating the layer-wise weights and activations and point out that the optimization could be made parallel by asynchronous update. Experiments on MNIST and CIFAR-10 datasets testify to the advantages of LPOM.

Hybrid backpropagation neural network-particle swarm optimization for seismic damage building prediction

The evaluation of the vulnerability of buildings to earthquakes is of prime importance to ensure a good plan can be generated for the disaster preparedness to civilians. Most of the attempts are directed in calculating the damage index of buildings to determine and predict the vulnerability to certain scales of earthquakes. Most of the solutions used are traditional methods which are time consuming and complex. Some of initiatives have proven that the artificial neural network methods have the potential in solving earthquakes prediction problems. However, these methods have limitations in terms of suffering from local optima, premature convergence and overfitting. To overcome this challenging issue, this paper introduces a new solution to the prediction on the seismic damage index of buildings with the application of hybrid back propagation neural network and particle swarm optimization (BPNN-PSO) method. The prediction was based on damage indices of 35 buildings around Malaysia. The BPNN-PSO demonstrated a better result of 89% accuracy compared to the traditional backpropagation neural network with only 84%. The capability of PSO supports fast convergence method has shown good effort to improve the processing time and accuracy of the results.

A new hyperparameters optimization method for convolutional neural networks

The use of convolutional neural networks involves hyperparameters optimization. Gaussian process based Bayesian optimization (GPEI) has proven to be an effective algorithm to optimize several hyperparameters. Then deep networks for global optimization algorithm (DNGO) that used neural network as an alternative to Gaussian process was proposed to optimize more hyperparameters.This paper presents a new algorithm that combines multiscale and multilevel evolutionary optimization (MSMLEO) with GPEI to optimize dozens of hyperparameters. These hyperparameters are divided into two groups. The first group related with the sizes of layers and kernels are discrete integers. The second group related with learning rates and so on is continuous floating-point numbers. All combinations of the first group are corresponding to the combinations of grid points on multi-scale grids and MSMLEO launches GPEI to optimize the second group of hyperparameters while the first group keeps fixed. The output of convolutional networks configured with above two groups of optimized hyperparameters is used as the fitness of MSMLEO. MSMLEO alternates with GPEI to search the optimal hyperparameters from coarsest scale to finest scale. Experimental results show that our algorithm has better performance and adaptability on optimizing dozens of hyperparameters of neural networks with a variety of numerical types.

A Wireless Localization Algorithm With BPNN-MEA-QPSO for Connected Vehicle

As one of key technologies of connected vehicles (CVs) applications, wireless localization can provide accurate and reliable vehicle location for high occupancy tolling and safety critical vehicle applications, such as collision avoidance. Several artificial intelligence methods, such as back propagation neural network (BPNN) and particle swarm optimization (PSO) method, have been employed to optimize the pass-loss model and to improve the accuracy of wireless localization algorithm. However, in view of the stochasticity of initial weights and thresholds in BPNN, it is difficult to reach the global convergence. In this study, a novel double-layer architecture for wireless localization algorithm is proposed based on the optimization of initial weights and thresholds in BPNN and the refinement of search direction and step in PSO algorithm. Based on the architecture, the wireless localization algorithm integrating BPNN with mind evolutionary algorithm (MEA) and quantum-behaved PSO (QPSO) method is proposed and validated using the experiment data in field environment. The validation results show that the proposed localization algorithm has better localization accuracy by comparison with the other localization algorithms, which the average error of the proposed localization algorithm in field environment is about 19 meters. In addition, the localization accuracy shows an improving tendency with the increasing of the number of base stations connected to moving vehicle. The location error is less than 10 meters when the number of base stations connected to the moving vehicle is greater than 7. For the wireless localization in field environment, the accuracy is acceptable for CVs applications.

Develop 24 dissimilar ANNs by suitable architectures & training algorithms via sensitivity analysis to better statistical presentation: Measure MSEs between targets & ANN for Fe–CuO/Eg–Water nanofluid

The artificial neural network optimization method is evaluated according to the experimental results of the hybrid non-Newtonian nanofluid of iron and copper oxide in a binary mixture of water and ethylene glycol concerned the mixture dynamic viscosity versus shear rate at different amounts of nanoparticles concentration and temperate. Present work novelty is demonstrated by providing 24 dissimilar ANN methods to introduce the suitable architectures and training algorithms for them. The mean squared errors (MSEs) between the targets and ANN outputs are evaluated to present the best optimization approach among them. Meanwhile the results would be supported by the appropriate sensitivity analysis to have better statistical visual presentation.

Optimization of neural networks via finite-value quantum fluctuations

We numerically test an optimization method for deep neural networks (DNNs) using quantum fluctuations inspired by quantum annealing. For efficient optimization, our method utilizes the quantum tunneling effect beyond the potential barriers. The path integral formulation of the DNN optimization generates an attracting force to simulate the quantum tunneling effect. In the standard quantum annealing method, the quantum fluctuations will vanish at the last stage of optimization. In this study, we propose a learning protocol that utilizes a finite value for quantum fluctuations strength to obtain higher generalization performance, which is a type of robustness. We demonstrate the performance of our method using two well-known open datasets: the MNIST dataset and the Olivetti face dataset. Although computational costs prevent us from testing our method on large datasets with high-dimensional data, results show that our method can enhance generalization performance by induction of the finite value for quantum fluctuations.

Parallel Implementation of the Nonlinear Semi-NMF Based Alternating Optimization Method for Deep Neural Networks

For computing weights of deep neural networks (DNNs), the backpropagation (BP) method has been widely used as a de-facto standard algorithm. Since the BP method is based on a stochastic gradient descent method using derivatives of objective functions, the BP method has some difficulties finding appropriate parameters such as learning rate. As another approach for computing weight matrices, we recently proposed an alternating optimization method using linear and nonlinear semi-nonnegative matrix factorizations (semi-NMFs). In this paper, we propose a parallel implementation of the nonlinear semi-NMF based method. The experimental results show that our nonlinear semi-NMF based method and its parallel implementation have competitive advantages to the conventional DNNs with the BP method.

Statistical Mechanical Analysis of Online Learning with Weight Normalization in Single Layer Perceptron

Weight normalization, a newly proposed optimization method for neural networks by Salimans and Kingma (2016), decomposes the weight vector of a neural network into a radial length and a direction v...

Generalized Sparse Recovery Model and Its Neural Dynamical Optimization Method for Compressed Sensing

This work investigates a sparse recovery minimization model and its neural network optimization method for the compressed sensing problem. One such nonsmooth and nonconvex model with \(l_1\)- and \(l_p\)-norms (\(1< p \le 2\)) is theoretically discussed for the uniqueness of its solutions with sparsity S under the restricted isometry property. A generalized gradient projection smoothing neural network based on smoothing approximation and gradient projection is designed to solve the model, due to the requirement of real-time problem solving. The existence, uniqueness and limit behavior of solutions of the neural network are well studied by means of the properties of gradient projection and function smoothness. Experimentally, the neural network is sufficiently examined, relying upon several state-of-the-art discrete numerical methods and neural network optimizers as well as multiple settings of p and different kinds of randomly generated sensing matrices. Numerical results have validated that the smaller/larger the p, the more effective the sparse recovery model under low/high coherent sensing matrices, and also that it can find sparse solutions and perform well over the compared neural networks and multiple discrete numerical solvers; especially, when the related sensing matrix with high coherence is non-RIP satisfying, it can recover the sparse signal with a high success rate.

The Influence of Volume Fraction of Martensite and Ferrite Grain Size on Ultimate Tensile Strength and Maximum Uniform True Strain of Dual Phase Steel

The aim of this study is to explain the contradiction of some results on interrelation of mechanical properties and microstructural analyses of dual phase (DP) steels. There have been some contradictions regarding the experimental results by different researchers about variation of ultimate tensile strength, and maximum uniform true strain of DP steels as functions of volume fraction of martensite. To justify these contradictions, a new model based on artificial neural network (ANN) optimization method has been proposed to predict the ultimate tensile strength, maximum uniform true strain and crash worthiness of dual phase steels versus volume fraction of martensite and grain size of ferrite. Results of ANN optimization model have been explained to analyze the effect of ferrite grain size on variation of mechanical properties.

Performance evaluation of two different neural network and particle swarm optimization methods for prediction of discharge capacity of modified triangular side weirs

Technical design of side weirs needs high accuracy in predicting discharge coefficient. In this study, discharge coefficient prediction performance of multi-layer perceptron neural network (MLPNN) and radial basis neural network (RBNN) were compared with linear and nonlinear particle swarm optimization (PSO) based equations. Performance evaluation of the model was done by using root mean squared error (RMSE), coefficient of determination (R2), mean absolute error (MAE), average absolute deviation (δ) and mean absolute relative error (MARE). Comparison of the results showed that both neural networks and PSO based equations could determine discharge coefficient of modified triangular side weirs with high accuracy. The RBNN with RMSE of 0.037 in test data was found to be better than MLPNN with RMSE of 0.044 and multiple linear and nonlinear PSO based equations (ML-PSO and MNL-PSO) with RMSE of 0.043 and 0.041, respectively. However, due to their simplicity, PSO based equations can be sufficient for use in practical cases.

Hybridization of Artificial Neural Network and Particle Swarm Optimization Methods for Time Series Forecasting

Recently, Particle Swarm Optimization (PSO) has evolved as a promising alternative to the standard backpropagation (BP) algorithm for training Artificial Neural Networks (ANNs). PSO is advantageous due to its high search power, fast convergence rate and capability of providing global optimal solution. In this paper, the authors explore the improvements in forecasting accuracies of feedforward as well as recurrent neural networks through training with PSO. Three widely popular versions of the basic PSO algorithm, viz. Trelea-I, Trelea-II and Clerc-Type1 are used to train feedforward ANN (FANN) and Elman ANN (EANN) models. A novel nonlinear hybrid architecture is proposed to incorporate the training strengths of all these three PSO algorithms. Experiments are conducted on four real-world time series with the three forecasting models, viz. Box-Jenkins, FANN and EANN. Obtained results clearly demonstrate the superior forecasting performances of all three PSO algorithms over their BP counterpart for both FANN as well as EANN models. Both PSO and BP based neural networks also achieved notably better accuracies than the statistical Box-Jenkins methods. The forecasting performances of the neural network models are further improved through the proposed hybrid PSO framework.

An Optimal State Feedback Controller Based Neural Networks for Synchronous Generator Excitation Control System

In this paper, an artificial neural network (ANN) is designed to optimize the matrix gain of state feedback controller. A linear mathematical model of a synchronous generator with excitation system is used as controlled system. The conventional methods that used to find the matrix gain need tedious calculations with compared to neural networks. The simulation proves that the proposed feedback controller based neural network optimization method has the better result in order to prove the dynamic performance of a single machine connected to infinite bus system(SMIB). The robustness of the proposed controller is tested by disturbance in excitation voltage. The results are compared with results of controller based on conventional methods. The potentials of the proposed technique are investigated using MATLAB software. DOI: http://dx.doi.org/10.11591/ijece.v3i4.3125

Optimal Power Control of Smart Grid Based on Neural Network Optimization

This paper presents a neural network based optimal control approach for the power allocation problem of smart grid. Firstly the optimal control framework which consists of center control layer, area control layer and generator control layer is presented. In order to reduce computation time of optimal control, the neural network optimization method and schematic diagram is proposed. Simulation is implemented in a modified IEEE30-bus power system comprising 6 conventional generators and a wind plant. Simulation results demonstrate that the proposed neural network optimization approach is satisfied the demand of computation time and the optimal power controller can allocate the power into loads reasonably and cope with the fluctuation of the load and renewable energy source’s output power.

Attribute dependency theory and its application on neural network

Neural network optimization methods are generally confined to learning algorithms and input attributes.Due to the high-dimensional mapping that neural network fits contains complex intrinsic attribute dependencies,the traditional optimization methods have not conducted the analytical study on it.The article put forward the attribute dependency theory based on functional dependency theory,elaborated the definition of the attribute dependency theory,and proved related theorem.Combining the Radius Basis Function(RBF)neural network,a new neural network optimization method based on attribute dependency theory(ADO-RBF)was proposed.

Numerical solution for high order differential equations using a hybrid neural network—Optimization method

This paper reports a novel hybrid method based on optimization techniques and neural networks methods for the solution of high order ordinary differential equations. Here neural networks is considered as a part of large field called neural computing or soft computing. This means that we propose a new solution method for the approximated solution of high order ordinary differential equations using innovative mathematical tools and neural-like systems of computation. This hybrid method can result in improved numerical methods for solving initial/boundary value problems, without using preassigned discretisation points. The mixture of feed forward neural networks and optimization techniques, based on Nelder–Mead method is used to introduce the close analytic form of the solution for the differential equation. Excellent test results are obtained for the solution of lower and higher order differential equations. The model finds approximation solution for the differential equation inside and outside the domain of consideration for the close enough neighborhood of initial/boundary points. Numerical examples are described to demonstrate the method.

Optimum surface roughness in end milling Inconel 718 by coupling neural network model and genetic algorithm

In this study, optimum cutting parameters of Inconel 718 are determined to enable minimum surface roughness under the constraints of roughness and material removal rate. In doing this, advantages of statistical experimental design technique, experimental measurements, artificial neural network and genetic optimization method are exploited in an integrated manner. Cutting experiments are designed based on statistical three-level full factorial experimental design technique. A predictive model for surface roughness is created using a feed forward artificial neural network exploiting experimental data. Neural network model and analytical definition of material removal rate are employed in the construction of optimization problem. The optimization problem was solved by an effective genetic algorithm for variety of constraint limits. Additional experiments have been conducted to compare optimum values and their corresponding roughness and material removal rate values predicted from the genetic algorithm. Generally a good correlation is observed between the predicted optimum and the experimental measurements. The neural network model coupled with genetic algorithm can be effectively utilized to find the best or optimum cutting parameter values for a specific cutting condition in end milling Inconel 718.

Fast Parameter Optimization of Large-Scale Electromagnetic Objects Using DIRECT with Kriging Metamodeling

With the advent of fast methods to significantly speed up numerical computation of large-scale realistic electromagnetic (EM) structures, EM design and optimization is becoming increasingly attractive. In recent years, genetic algorithms, neural network and evolutionary optimization methods have become increasingly popular for EM optimization. However, these methods are usually associated with a slow convergence bound and, furthermore, may not yield a deterministic optimal solution. In this paper, a new hybrid method using Kriging metamodeling in conjunction with the divided rectangles (DIRECT) global-optimization algorithm is used to yield a globally optimal solution efficiently. The latter yields a deterministic answer with fast convergence bounds and inherits both local and global-optimization properties. Three examples are given to illustrate the applicability of the method, Le., shape optimization for a slot-array frequency-selective surface, antenna location optimization to minimize EM coupling from the antenna to RF devices in automobile structures, and multisensor optimization to satisfy RF coupling constraints on a vehicular chassis in the presence of a wire harness. In the first example, DIRECT with Kriging surrogate modeling was employed. In the latter two examples, the adaptive hybrid optimizer, superEGO, was used. In all three examples, emphasis is placed on the speed of convergence, as well as on the flexibility of the optimization algorithms.

Autonomous Autorotation Landing of Small Unmanned Helicopter

In this paper, we derive a linear and a nonlinear model of helicopter autorotation maneuver focusing on main rotor revolution and decent rate dynamics. The nonlinear model for above dynamics is derived from Blade Element Theory combined with the induced velocity linear approximation proposed in this paper. The linear model is derived by implementing a neural network optimization method based on experimental data. The derived nonlinear and linear models are verified by comparing simulated data of the model with experimental one. Finally we carry out preliminary experiment of autorotation landing with a simple PI controller. Rotor revolution and decent rate are found to be well controlled by the designed controller.

Differential Evolution Training Algorithm for Feed-Forward Neural Networks

An evolutionary optimization method over continuous search spaces, differential evolution, has recently been successfully applied to real world and artificial optimization problems and proposed also for neural network training. However, differential evolution has not been comprehensively studied in the context of training neural network weights, i.e., how useful is differential evolution in finding the global optimum for expense of convergence speed. In this study, differential evolution has been analyzed as a candidate global optimization method for feed-forward neural networks. In comparison to gradient based methods, differential evolution seems not to provide any distinct advantage in terms of learning rate or solution quality. Differential evolution can rather be used in validation of reached optima and in the development of regularization terms and non-conventional transfer functions that do not necessarily provide gradient information.