Optimization Of Network Weights Research Articles

Wind power forecasting based on ensemble deep learning with surrogate-assisted evolutionary neural architecture search and many-objective federated learning

Deep neural networks (DNN) have been widely used in wind power forecasting (WPF), however, they still encounter difficulties such as delayed variable selection, DNN architecture design, and network weight optimization. Therefore, an ensemble cascaded DNN (ECDNN) with surrogate-assisted evolutionary neural architecture search (SEANS) and many-objective federated learning (MOFL) is proposed and referred to as ECDNN-SENAS-MOFL. SEANS is proposed to determine the delayed variables and network architecture efficiently, which allows significantly enhancing the evolutionary search efficiency than traditional evolutionary neural architecture search (ENAS) approaches by employing a surrogate model to achieve fast fitness evaluations. Meanwhile, MOFL is developed by embedding the federated learning into many-objective optimization to obtain more reliable network weights based on the distributed modeling data than traditional centralized learning and federated learning, as well as solving the problems of data privacy protection and data islands. Furthermore, selective ensemble learning is introduced to build high-performance ensemble WPF model by combining the merits of multiple highly influential Pareto solutions. The effectiveness and superiority of the proposed ECDNN-SENAS-MOFL method are verified using an actual wind power dataset. The application results show that the proposed method can provide new insights into data-driven WPF modeling and has a great potential for practical applications.

Ultra-short-term wind power probabilistic forecasting based on an evolutionary non-crossing multi-output quantile regression deep neural network

Ultra-short-term wind power probabilistic forecasting is of significance for stable power grid operation; however, it is still challenging due to the inherent nonlinearity and uncertainty. Most state-of-the-art methods have focused on achieving quantile prediction using a combination of linear quantile regression and nonlinear complex deep neural networks. Yet, these methods struggle with several dilemmas. Quantile regression deep neural networks require a complete training once for each quantile. The multi-training mode and complex structure of quantile regression deep neural network can lead to extremely high computational complexity. Most of the training of quantile regression deep neural networks are guided by the loss of each quantile, and the weights are adjusted by gradient descent in which the gradient explosion and quantile crossover may be encountered. Therefore, this paper proposes a non-crossing multi-output quantile regression deep neural network optimized by chaotic particle swarm optimization. It designs a multi-output deep neural network to output all quantile estimations simultaneously through one training, effectively solving the structural complexity problem of traditional quantile regression deep neural networks. Since quantile regression produces a non-differentiable loss function which significantly hinders model training, the proposed neural network is trained by chaotic particle swarm optimization. It not only achieves the effect of optimizing all quantile losses simultaneously, but also can significantly alleviate the dilemma of training in traditional neural network weight optimization. In addition, several non-crossing constraints are designed for avoiding quantile crossover. The proposed model is trained and tested on two real-world wind power case studies. The experiment results show that the proposed model shows superiority in performance criteria, training speed, and avoiding quantile crossover.

Sparse neural network optimization by Simulated Annealing

The over-parameterization of neural networks and the local optimality of backpropagation algorithm have been two major problems associated with deep-learning. In order to reduce the redundancy of neural network parameters, the conventional approach has been to prune branches with small weights. However, this only solves the problem of parameter redundancy, not providing any global optimality guarantees. In this paper, we overturn back-propagation and combine the sparse network optimization problem and the network weight optimization problem using a non-convex optimization method, namely Simulated Annealing. This method can complete network training under the premise of controlling the amount of parameters. Different from simply updating network parameters using gradient descent, our method simultaneously optimizes the topology of the sparse network. With the guarantee of global optimality of Simulated Annealing solution, the performance of the sparse network optimized by our method has exceeded the one trained by backpropagation only.

A Lightweight Detection Method for Remote Sensing Images and Its Energy-Efficient Accelerator on Edge Devices.

Convolutional neural networks (CNNs) have been extensively employed in remote sensing image detection and have exhibited impressive performance over the past few years. However, the abovementioned networks are generally limited by their complex structures, which make them difficult to deploy with power-sensitive and resource-constrained remote sensing edge devices. To tackle this problem, this study proposes a lightweight remote sensing detection network suitable for edge devices and an energy-efficient CNN accelerator based on field-programmable gate arrays (FPGAs). First, a series of network weight reduction and optimization methods are proposed to reduce the size of the network and the difficulty of hardware deployment. Second, a high-energy-efficiency CNN accelerator is developed. The accelerator employs a reconfigurable and efficient convolutional processing engine to perform CNN computations, and hardware optimization was performed for the proposed network structure. The experimental results obtained with the Xilinx ZYNQ Z7020 show that the network achieved higher accuracy with a smaller size, and the CNN accelerator for the proposed network exhibited a throughput of 29.53 GOPS and power consumption of only 2.98 W while consuming only 113 DSPs. In comparison with relevant work, DSP efficiency at an identical level of energy consumption was increased by 1.1-2.5 times, confirming the superiority of the proposed solution and its potential for deployment with remote sensing edge devices.

In-Depth Case Study on Artificial Neural Network Weights Optimization Using Meta-Heuristic and Heuristic Algorithmic Approach

The Meta-heuristic and Heuristic algorithms that have been introduced for deep neural network optimization is in this paper. Artificial Intelligence, and also the most used Deep Learning methods are all growing in popularity these days, thus we need faster optimization strategies for finding the results of future activities. Neural Network Optimization with Particle Swarm Optimization, Backpropagation (BP), Resilient Propagation (Rprop), and Genetic Algorithm (GA) is used for numerical analysis of different datasets and comparing each other to find out which algorithms work better for finding optimal solutions by reducing training loss. Genetic algorithm and also bio-inspired Particle Swarm Optimization is introduced in this paper. Besides, Resilient Propagation and Conventional Backpropagation algorithms which are application-specific algorithms have also been introduced. Meta-heuristic algorithms GA and PSO are a higher-level formula and problem-independent technique that may be used to a diverse number of challenges. The characteristic of Heuristic algorithms has extremely specific features that vary depending on the problem. The conventional Backpropagation (BP) based optimization, the Particle Swarm Optimization methodology, and Resilient Propagation (Rprop) are all fully presented, and how to apply these procedures in Artificial Deep Neural networks Optimization is also thoroughly described. Applied numerical simulation over several datasets proves that the Meta-heuristic algorithm Particle Swarm Optimization and also Genetic Algorithm performs better than the conventional heuristic algorithm like Backpropagation and Resilient Propagation.

Noise-Adaptive Non-Blind Image Deblurring.

This work addresses the problem of non-blind image deblurring for arbitrary input noise. The problem arises in the context of sensors with strong chromatic aberrations, as well as in standard cameras, in low-light and high-speed scenarios. A short description of two common classical approaches to regularized image deconvolution is provided, and common issues arising in this context are described. It is shown how a pre-deconvolved deep neural network (DNN) based image enhancement can be improved by joint optimization of regularization parameters and network weights. Furthermore, a two-step approach to deblurring based on two DNNs is proposed, with the first network estimating deconvolution regularization parameters, and the second one performing image enhancement and residual artifact removal. For the first network, a novel RegParamNet architecture is introduced and its performance is examined for both direct and indirect regularization parameter estimation. The system is shown to operate well for input noise in a three orders of magnitude range (0.01–10.0) and a wide spectrum of 1D or 2D Gaussian blur kernels, well outside the scope of most previously explored image blur and noise degrees. The proposed method is found to significantly outperform several leading state-of-the-art approaches.

Benchmarking Studies Aimed at Clustering and Classification Tasks Using K-Means, Fuzzy C-Means and Evolutionary Neural Networks

Clustering is a widely used unsupervised learning technique across data mining and machine learning applications and finds frequent use in diverse fields ranging from astronomy, medical imaging, search and optimization, geology, geophysics, and sentiment analysis, to name a few. It is therefore important to verify the effectiveness of the clustering algorithm in question and to make reasonably strong arguments for the acceptance of the end results generated by the validity indices that measure the compactness and separability of clusters. This work aims to explore the successes and limitations of two popular clustering mechanisms by comparing their performance over publicly available benchmarking data sets that capture a variety of data point distributions as well as the number of attributes, especially from a computational point of view by incorporating techniques that alleviate some of the issues that plague these algorithms. Sensitivity to initialization conditions and stagnation to local minima are explored. Further, an implementation of a feedforward neural network utilizing a fully connected topology in particle swarm optimization is introduced. This serves to be a guided random search technique for the neural network weight optimization. The algorithms utilized here are studied and compared, from which their applications are explored. The study aims to provide a handy reference for practitioners to both learn about and verify benchmarking results on commonly used real-world data sets from both a supervised and unsupervised point of view before application in more tailored, complex problems.

Botnet Attack Detection Using Local Global Best Bat Algorithm for Industrial Internet of Things

The need for timely identification of Distributed Denial-of-Service (DDoS) attacks in the Internet of Things (IoT) has become critical in minimizing security risks as the number of IoT devices deployed rapidly grows globally and the volume of such attacks rises to unprecedented levels. Instant detection facilitates network security by speeding up warning and disconnection from the network of infected IoT devices, thereby preventing the botnet from propagating and thereby stopping additional attacks. Several methods have been developed for detecting botnet attacks, such as Swarm Intelligence (SI) and Evolutionary Computing (EC)-based algorithms. In this study, we propose a Local-Global best Bat Algorithm for Neural Networks (LGBA-NN) to select both feature subsets and hyperparameters for efficient detection of botnet attacks, inferred from 9 commercial IoT devices infected by two botnets: Gafgyt and Mirai. The proposed Bat Algorithm (BA) adopted the local-global best-based inertia weight to update the bat’s velocity in the swarm. To tackle with swarm diversity of BA, we proposed Gaussian distribution used in the population initialization. Furthermore, the local search mechanism was followed by the Gaussian density function and local-global best function to achieve better exploration during each generation. Enhanced BA was further employed for neural network hyperparameter tuning and weight optimization to classify ten different botnet attacks with an additional one benign target class. The proposed LGBA-NN algorithm was tested on an N-BaIoT data set with extensive real traffic data with benign and malicious target classes. The performance of LGBA-NN was compared with several recent advanced approaches such as weight optimization using Particle Swarm Optimization (PSO-NN) and BA-NN. The experimental results revealed the superiority of LGBA-NN with 90% accuracy over other variants, i.e., BA-NN (85.5% accuracy) and PSO-NN (85.2% accuracy) in multi-class botnet attack detection.

Topology Variations of an Amplifier-based MOS Analog Neural Network Implementation and Weights Optimization

Neural networks are achieving state-of-the-art performance in many applications, from speech recognition to computer vision. A neuron in a multi-layer network needs to multiply each input by its weight, sum the results and perform an activation function. This paper is an extended version of the article in which we present an implementation of an amplifier-based MOS analog neuron and the optimization of the synaptic weights using in-loop circuit simulations. In addition to the base topology, we present two variations of the original conference paper topology to reduce area and power. MOS transistors operating in the triode region are used as variable resistors to convert the input and weight voltage to proportional input current. To test the analog neuron in full networks, an automatic generator is developed to produce a netlist based on the number of neurons on each layer, inputs, and weights. Simulation results using a CMOS 180 nm technology for all topologies demonstrate the neuron proper transfer function and its functionality while trained in test datasets.

Cloud Education Chain and Education Quality Evaluation Based on Hybrid Quantum Neural Network Algorithm

This paper proposes the functional model and application service implementation process of the education cloud platform application service architecture. The entire cloud application service architecture mainly includes four parts: cloud service management, cloud application service rapid creation and deployment, dynamic process configuration, and unified identity authentication. Based on the basic theory of workflow, the process status and business services of cloud application services are discussed. The BP neural network weight optimization model based on the improved quantum evolution method is studied, and a method that combines the improved quantum evolution algorithm (IQEA) and the BP algorithm to complete the back propagation neural network training is proposed, that is, the IQEA‐BP algorithm. Firstly, the traditional quantum evolution algorithm is improved, and then, the improved quantum evolution algorithm is used to optimize the network weights as a whole to overcome the shortcomings of the BP algorithm that is easy to fall into the local optimum; then, we use the BP algorithm to find the better weight as the initial value to improve the training and prediction accuracy of the network. In order to enrich the school education quality evaluation system, this article adds soft indicators that can reflect school education performance on the basis of the existing “National Education Inspection Team” indicators and uses analytical methods to prove the effectiveness and feasibility of the new evaluation indicators. The X1‐X10 index data is selected as the evaluation index of the school education quality evaluation system in this paper. Testing the performance of the BP neural network, the accuracy rate of the school education quality evaluation is 93.3%, the average absolute error is 0.067, and the accuracy and recall rate of the test set grade gradient of 0, 1, 2, 3, 5, 6, and 8 are all 93%, indicating that the IQEA‐BP neural network algorithm has a good effect on the evaluation of school education quality.

Global-best optimization of ANN trained by PSO using the non-extensive cross-entropy with Gaussian gain

An efficient optimization of network weights has been the primary goal of the artificial neural network (ANN) research community since decades. The aim of every optimization problem is to minimize the network cost which is some form of error function between the desired and the actual network outputs, during the training phase. The conventional gradient-based optimization algorithms like backpropagation are likely to get trapped in local minima and are sensitive to choices of initial weights. The evolutionary algorithms have proved their usefulness in introducing randomness into the optimization procedure, since they work on a global search strategy and induce a globally minimum solution for the network weights. In this paper, we particularly focus on ANN trained by Particle Swarm Optimization (ANN-PSO), in which the local-best and global-best particle positions represent possible solutions to the set of network weights. The global-best position of the swarm, which corresponds to the minimum cost function over time, is determined in our work by minimizing a new non-extensive cross-entropy error cost function. The non-extensive cross-entropy is derived from the non-extensive entropy with Gaussian gain that has proven to give minimum values for regular textures containing periodic information represented by uneven probability distributions. The new cross-entropy is defined, and its utility for optimizing the network weights to a globally minimum solution is analyzed in this paper. Extensive experimentation on two different versions: the baseline ANN-PSO and one of its most recent variants IOPSO-BPA, on benchmark datasets from the UCI repository, with comparisons to the state of the art, validates the efficacy of our method.

Modeling of the Hot Flow Behaviors for Ti-6Al-4V-0.1Ru Alloy by GA-BPNN Model and Its Application

AbstractA series of compression tests were performed on Ti-6Al-4V-0.1Ru titanium alloy in nine temperatures between 750 and 1150 °C and a strain rate range of 0.01 to 10s−1. The hot deformation behaviors of Ti-6Al-4V-0.1Ru showed highly non-linear intrinsic relationships with temperature, strain and strain rate. The flow curves exhibited different softening mechanisms, dynamic recrystallization (DRX) and dynamic recovery (DRV). In this study, the rheological behaviors of Ti-6Al-4V-0.1Ru were modeled using a special hybrid prediction model, where genetic algorithm (GA) was implemented to do a back-propagation neural network (BPNN) weights optimization, namely GA-BPNN. Subsequently, the predicted results were compared with experimental values and GA-BPNN model showed the ability to predict the flow behaviors of Ti-6Al-4V-0.1Ru with superior accuracy. Then a 3-D continuous interaction space was constructed to visually reveal the successive relationships among processing parameters. Finally, the predicted data were applied to process simulation and accuracy results were achieved.

A data-parallelism approach for PSO-ANN based medical image reconstruction on a multi-core system

Abstract This paper presents the sequential and parallel data decomposition strategies implemented on a Particle Swarm Optimization (PSO) algorithm based Artificial Neural Network (PSO-ANN) weights optimization for image reconstruction. The application system is developed for the reconstruction of two-dimensional spatial standard Computed Tomography (CT) phantom images. It is running on a multi-core computer by varying the number of cores. The feed forward ANN initializes the weight between the ‘ideal’ images that are reconstructed using filtered back projection (FBP) technique and the corresponding projection data of CT phantom. In an earlier work, ANN training time is too long. Hence, we propose that the ANN exemplar datasets are decomposed into subsets. Using these subsets, artificial sub neural nets (subnets) are initialized and each subnet initial weights are optimized using PSO. Consequently, it was observed that the sequential approach of the proposed method consumes more training time. Hence the parallel strategy is attempted to reduce the computational training time. The parallel approach is further explored for image reconstruction from ‘noisy’ and ‘limited-angle’ datasets also.

Artificial Neural Network Weight Optimization: A Review

Artificial Neural Network Weight Optimization: A Review

Artificial neural network weights optimization based on social-based algorithm to realize sediment over the river

Predictions of sediment load are required in a wide spectrum of problems such as design of the dead volume of a dam, sediment transport in the river, design of stable channels, and dredging needs. Researchers have used the regression between sediment concentration and water discharge. Such relationships are obtained through the application of regression analysis in many studies. Unfortunately, in the classical regression approach to determine sediment concentration---water discharge relationships, internal uncertainties are not taken explicitly into consideration. Therefore, researchers look for non-linear methods to estimate sediment load such as artificial neural network (ANN) methods to solve non-linear problems. The main purpose of this paper is to optimize the ANN connection weights with novel social-based algorithm (SBA) to realize the sediment in Maroon river. The SBA tries to capture several people in different types of countries. They try to reach high levels. The approach illustrated feed-forward neural network optimization for sediment estimation of four Maroon river stations in Iran, which was called FF-SBA. Three inputs were presented in each station: length of river, discharge and debit. Sediment parameter on the same station is measured as the output parameter. Results of optimization algorithms such as the genetic algorithm, particle swarm optimization and imperialist competitive algorithm were compared with the SBA results, and it was found that the FF-SBA model exhibited more capability, flexibility, and accuracy in sediment training, testing, and forecasting steps for the Maroon river in Iran.

Multiple Quantum Spaces Based Genetic Coding Method

Quantum genetic coding method plays an important role in improving the efficiency of optimization algorithm. The existing quantum genetic algorithm has some defects, that quantum encoding scheme is tend to reduce the stability, so that the algorithm is prone to premature convergence and falls into local minima. Therefore, the chain of multiple genes encoding scheme is used to extend in the multi-dimensional space for improving this algorithm. By function extremum and simulation of neural network weights optimization, according to the characteristics of qubits and the normalization condition, double and triple chain binding coding schemes are proposed. By experiments on multiple genes encoding scheme chain, the performance of the algorithm is tested. It shows that the algorithm can get better results by increasing the higher accuracy of solution chain genes. It is an effective strategy to improve the performance of genetic coding.

Differential Evolution and Levenberg Marquardt Trained Neural Network Scheme for Nonlinear System Identification

This paper proposes a new nonlinear system identification scheme using differential evolution (DE), neural network and Levenberg Marquardt algorithm (LM). Here, DE and LM in a combined framework are used to train a neural network for achieving better convergence of neural network weight optimization. A number of examples including a practical case-study have been considered for implementation of different system identification methods namely, only NN, DE+NN and DE+LM+NN. After, a series of simulation studies of these methods on the different nonlinear systems it has been confirmed that the proposed DE and LM trained NN approach to nonlinear system identification has yielded better identification results in terms of time of convergence and less identification error.

Quantum-inspired evolutionary algorithm for continuous space optimization based on Bloch coordinates of qubits

A novel quantum-inspired evolutionary algorithm is proposed based on the Bloch coordinates of quantum bits (qubits) in this paper. The chromosome is comprised of qubits whose Bloch coordinates comprise gene chain. The quantum chromosomes are updated by quantum rotation gates, and are mutated by quantum non-gates. For the rotation direction of quantum rotation gates, a simple determining method is proposed. For the rotation and mutation of qubits, two new operators are constructed based on Bloch coordinates of qubits. In this algorithm, the Bloch coordinates of each qubit are regarded as three paratactic genes, each chromosome contains three gene chains, and each gene chain represents an optimization solution, which can accelerate the convergence process for the same number of chromosomes. By two application examples of function extremum and neural network weights optimization, the simulation results show that the approach is superior to common quantum evolutionary algorithm and simple genetic algorithm in both search capability and optimization efficiency.

Evapotranspiration modelling from climatic data using a neural computing technique

AbstractEvapotranspiration is one of the basic components of the hydrologic cycle and essential for estimating irrigation water requirements. This paper investigates the modelling of evapotranspiration using the feed‐forward artificial neural network (ANN) technique with the Levenberg–Marquardt (LM) training algorithm. The LM algorithm has never been used in evapotranspiration estimation before. The LM is used for the optimization of network weights, since this algorithm is more powerful and faster than the conventional gradient descent. Various combinations of daily climatic data, i.e. wind speed, air temperature, relative humidity and solar radiation, from three stations in Los Angeles, USA, are used as inputs to the ANN so as to evaluate the degree of effect of each of these variables on evapotranspiration. A comparison is made between the estimates provided by the ANN and those of the following empirical models: Penman, Hargreaves, Turc. Mean square error, mean absolute error and determination coefficient statistics are used as comparing criteria for the evaluation of the models' performances. Based on the comparisons, it was found that the neural computing technique could be employed successfully in modelling evapotranspiration process from the available climatic data. The results also indicate that the Hargreaves method provides better performance than the Penman and Turc methods in estimation of the evapotranspiration. The accuracy of the ANN technique in evapotranspiration estimation using nearby station data was also investigated. Copyright © 2007 John Wiley & Sons, Ltd.

Genetic reinforcement learning for neurocontrol problems

Empirical tests indicate that at least one class of genetic algorithms yields good performance for neural network weight optimization in terms of learning rates and scalability. The successful application of these genetic algorithms to supervised learning problems sets the stage for the use of genetic algorithms in reinforcement learning problems. On a simulated inverted-pendulum control problem, “genetic reinforcement learning” produces competitive results with AHC, another well-known reinforcement learning paradigm for neural networks that employs the temporal difference method. These algorithms are compared in terms of learning rates, performance-based generalization, and control behavior over time.