Convergence Of Neural Network Research Articles

Convergence of multiple deep neural networks for classification with fewer labeled data

With the advent of deep neural networks (DNNs) in the last two decades, tremendous developments have been made in many fields, such as image classification/recognition, voice recognition, and action recognition. These advanced DNNs require large amounts of labeled data, whose collection is costly and requires great effort. In this paper, we provide a convergence method for DNNs to solve some of these difficulties. First, we consider how to create labeled data using a generative adversarial network (GAN), one DNN method, and add additional networks to improve the quality of generated data. Then, we propose a convergence method for the DNNs and use a three-step evaluation to confirm this approach and show how to use the automatically generated data for training. With the method proposed in this paper, we hope that the manual work of labeling data can be reduced for many DNN applications.

Complex-Valued Convolutional Neural Networks Design and its Application on UAV DOA Estimation in Urban Environments

Direction-of-arrival (DOA) estimation is an important task in many unmanned aerial vehicle (UAV) applications. However, the complicated electromagnetic wave propagation in urban environments substantially deteriorates the performance of many conventional model-driven DOA estimation approaches. To alleviate this, a deep learning based DOA estimation approach is proposed in this paper. Specifically, a complex-valued convolutional neural network (CCNN) is designed to fit the electromagnetic UAV signal with complex envelope better. In the CCNN design, we construct some mapping functions using quantum probabilities, and further analyze some factors which may impact the convergence of complex-valued neural networks. Numerical simulations show that the proposed CCNN converges faster than the real convolutional neural network, and the DOA estimation result is more accurate and robust.

Adaptive neural impedance control with extended state observer for human–robot interactions by output feedback through tracking differentiator

Although impedance control has huge application potential in human–robot cooperation, its engineering application is still quite limited, owing to the high nonlinearity of the human–robot dynamics and disturbances. This article presents a novel adaptive neural network controller with extended state observer for the human–robot interaction using output feedback. The adaptive neural network with extended state observer integrates the adaptive neural network and extended state observer to combine their advantages. The proposed algorithm can address the challenges encountered in human–machine systems, for example, slow convergence of neural networks, internal and external disturbances. Output feedback is realized using tracking differentiator to avoid the costly measurements of certain states. The errors of the closed-loop system are proven to converge to a small compact set containing 0 by Lyapunov theory. Simulations and experiments were conducted to verify the effectiveness of the proposed controller. Results show that the proposed strategy offers superior convergence and better tracking performance compared with the adaptive neural network. The proposed controller can be widely applied in various human–machine interactions to enhance productivity and efficiency.

Parameter Estimation for PMSM based on a Back Propagation Neural Network Optimized by Chaotic Artificial Fish Swarm Algorithm

Permanent Magnet Synchronous Motor(PMSM) control system with strong nonlinearity makes it difficult to accurately identify motor parameters such as stator winding, dq axis inductance, and rotor flux linkage. Aiming at the premature convergence of traditional Back Propagation Neural Network(BPNN) in PMSM motor parameter identification, a new method of PMSM motor parameter identification is proposed. It uses Chaotic Artificial Fish Swarm Algorithm(CAFSA) to optimize the initial weights and thresholds of BPNN, and then strengthens training by BPNN algorithm. Thus, the global optimal network parameters are obtained by using the global optimization of CAFSA and the local search ability of BPNN. The simulation results and experimental data show that the initial value sensitivity of the network model optimized by CAFS-BPNN Algorithm is weak, the parameter setting is robust, and the system stability is good under complex conditions. Compared with other intelligent algorithms, such as RSL and PSO, CAFS-BPNNA has high identification accuracy and fast convergence speed for PMSM motor parameters.

An Improved DDPG and Its Application Based on the Double-Layer BP Neural Network

This paper focused on three application problems of the traditional Deep Deterministic Policy Gradient(DDPG) algorithm. That is, the agent exploration is insufficient, the neural network performance is unsatisfied, the agent output fluctuates greatly. In terms of agent exploration strategy, network training algorithm and overall algorithm implementation, an improved DDPG method based on double-layer BP neural network is proposed. This method introduces fuzzy algorithm and BFGS algorithm based on Armijo-Goldstein criterion, improves the exploration efficiency, learning efficiency and convergence of BP neural network, increases the number of layers of BP neural network to improve the fitting ability of the network, and adopts periodic update to ensure the stable operation of the algorithm. The experimental results show that the deep learning network based on the improved DDPG algorithm has greatly improved the performance compared with the traditional method after multiple rounds of self-learning under variable working conditions. This study lays a theoretical and experimental foundation for the extended application of deep learning algorithm.

Improved Artificial Bee Colony Algorithm Applied to Neural Network Photovoltaic Power Generation Prediction Method

With the increase in the use of renewable energy, especially the development and utilization of solar energy resources, accurate photovoltaic power generation prediction technology will help the promotion of photovoltaic power generation. The amount of photovoltaic power generation depends on weather conditions, and it is easy to produce large fluctuations under different weather conditions. Its power generation has the characteristics of randomness, fluctuation and intermittency. In view of the shortcomings of the traditional BP neural network prediction method, this paper proposes an improved artificial bee colony algorithm. The improved artificial bee colony algorithm is used to optimize the network parameter weights in the traditional BP algorithm, and the two algorithms are merged in global iteration. Based on the characteristics of training light intensity, weather, temperature and historical power value of photovoltaic output power,a photovoltaic power generation prediction model is established. The simulation results show that the improved artificial bee colony algorithm in the neural network’s photovoltaic power generation forecast improves the accuracy and convergence speed of the traditional BP neural network convergence solution, and can provide more comprehensive information for grid power dispatch and control.

Adaptive activation functions accelerate convergence in deep and physics-informed neural networks

We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima. Furthermore, the proposed adaptive activation functions are shown to accelerate the minimization process of the loss values in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion datasets with and without data augmentation.

Architectural Parameter-Independent Network Initialization Scheme for Sigmoidal Feedforward ANNs

The selection of the initial network weights has been a known key aspect affecting the convergence of sigmoidal activation function-based artificial neural networks. In this paper, a new network initialization scheme has been proposed that initializes the network weights such that activation functions in the network are not saturated initially. The proposed method ensures that the initial outputs of the hidden neurons are in the active region which positively impacts the network’s rate of convergence. Unlike most of the earlier proposed initialization schemes, this method does not depend on architectural parameters like the size of the input layer or the hidden layer. The performance of the proposed scheme has been compared against eight well-known weight initialization routines over six benchmark real-world problems. Results show that the proposed weight initialization routine enables the network to achieve better performance within the same count of network training epochs. A right-tailed t-test also shows that our proposed scheme is significantly better in most of the cases against the other techniques or statistically similar in a few cases but never underperforms. Hence, it may be considered as a strong alternative to the conventional neural network initialization techniques.

Prediction model for the water jet falling point in fire extinguishing based on a GA-BP neural network.

Past research on the process of extinguishing a fire typically used a traditional linear water jet falling point model and the results ignored external factors, such as environmental conditions and the status of the fire engine, even though the water jet falling point location prediction was often associated with these parameters and showed a nonlinear relationship. This paper constructed a BP (Back Propagation) neural network model. The fire gun nozzle characteristics were included as model inputs, and the water discharge point coordinates were the model outputs; thus, the model could precisely predict the water discharge point with small error and high precision to determine an accurate firing position and allow for the timely adjustment of the spray gun. To improve the slow convergence and local optimality problems of the BP neural network (BPNN), this paper further used a genetic algorithm to optimize the BPNN (GA-BPNN). The BPNN can be used to optimize the weights in the network to train them for global optimization. A genetic algorithm was introduced into the neural network approach, and the water jet landing prediction model was further improved. The simulation results showed that the prediction accuracy of the GA-BP model was better than that of the BPNN alone. The established model can accurately predict the location of the water jet, making the prediction results more useful for firefighters.

Composite adaptive control with fast convergence for multilayer neural network

SummaryA composite adaptive control (CAC) that combines the benefits of direct and indirect adaptive controls has better parameter adaptation and control response. Multilayer neural networks (NNs) can be employed to enhance a model's representation capacity, but previous composite adaptive approaches cannot easily train the model due to its nonlinearities. A novel CAC is therefore developed in this study to tackle the above limitations. A modified robust version is adopted by focusing on the direct adaptive part to enhance robustness of adaption. Then, the indirect parameter adaptive law is improved by adopting a small learning rate in which a multistep adaption update is executed in one control interval. Moreover, multistep prediction errors are implemented to guarantee the consistency of the approximation errors, and an experience replay technique is adopted to attenuate the requirement of persistent excitation conditions. These improvements not only accelerate the convergence process but also smoothen the updating of NN parameters. Given that a nonlinear plant with MIMO strict‐feedback structure is considered, the proposed CAC is integrated into the backstepping framework. The uniformly bounded property of the tracking errors and the approximation errors is proven by Lyapunov theory. The superiority of the proposed method and the roles of these improvements are demonstrated by comparative simulations.

Determination of Lycopersicon maturity using convolutional autoencoders

The field of computer science is witnessing the development of new and advanced applications in agricultural science and related technologies. Accurate evaluations of the ripeness of fruits and vegetables are very important in agricultural science as fruit growers can profit from the automatic detection and interpretation of fruit maturity levels. In this study, we propose a method of classifying Lycopersicons based on three maturity levels (immature, semi-mature, and mature). Our method includes two artificial neural networks, a convolutional autoencoder (CAE), and a backpropagation neural network with a Softmax layer. A CAE involves the convergence of convolutional neural networks and an autoencoder and has recently gained considerable attention in the field of Engineering. However, a traditional backpropagation neural network also plays an important role in the proposed method. To adapt the classification system to various complex scenarios, the CAE functions as a background filter, and it determines the region of interest (ROI) in an image. A contribution of this study is the use of a CAE to determine the ROI in the Lycopersicon image instead of tuning handcrafted parameters manually to set the ROI. The machine detects the Lycopersicon through self-learning mechanisms. Using the extracted features, the machine employs self-learning mechanisms to determine Lycopersicon maturity. The experimental results demonstrate that our method can recognize maturity levels with an accuracy rate of 100%. Therefore, the proposed algorithm provides objective and useful information concerning maturity to optimize the harvest time of Lycopersicons.

Classifcation of events in information security systems based on neural networks

Purpose of the research.The aim of the study is to increase the effectiveness of information security and to enhance accuracy and promptness of the classification of security events, security incidents, and threats in information security systems. To respond to this challenge, neural network technologies were suggested as a classification tool for information security systems. These technologies allow accommodating incomplete, inaccurate and unidentified raw data, as well as utilizing previously accumulated information on security issues. To address the problem more effectively, collective methods based on collective neural ensembles aligned with an advanced complex approach were implemented.Materials and methods:When solving complex classification problems, often none of the classification algorithms provides the required accuracy. In such cases, it seems reasonable to build compositions of algorithms, mutually compensating errors of individual algorithms. The study also gives an insight into the application of neural network ensemble to address security issues in the corporate information system and provides a brief review of existing approaches to the construction of neural network ensembles and methods to shape problem solving with neural networks classifiers. An advanced integrated approach is proposed to tackle problems of security event classification based on neural network ensembles (neural network committees). The approach is based on a three-step procedure. The stages of the procedure implementation are described. It is shown that the use of this approach facilitates the efficiency of solving the problem.Results:An advanced integrated approach to addressing security event classification based on neural network ensembles (neural network committees) is proposed. This approach applies adaptive reduction of neural network ensemble (selection of the best classifiers is based on the assessment of the compliance degree of the competence area of the private neural network classifier and convergence of the results of private classifiers), as well as the selection and rationale of the voting method (composition or aggregation of outputs of private classifiers). The results of numerical experiments support the effectiveness of the proposed approach.Conclusion:Collectively used artificial neural networks in the form of neural network ensembles (committees of neural networks) will provide more accurate and reliable results of security event classification in the corporate information network. Moreover, an advanced integrated approach to the construction of a neural network ensemble is proposed to facilitate effectiveness of the classification process. The approach is based on the application of the adaptive reduction procedure for the results of private classifiers and the procedure for selecting the method of aggregation of the results of private classifiers. These outcomes will enable advancement of the system control over information security incidents. Finally, the paper defines tendencies and directions of the development of collective solution methods applying neural network ensembles (committees of neural networks).

The effect of Z-Score standardization (normalization) on binary input due the speed of learning in back-propagation neural network

The speed of learning in neural network environment is considered as the most effective parameter spatially in large data sets. This paper tries to minimize the time required for the neural network to fully understand and learn about the data by standardize input data. The paper showed that the Z-Score standardization of input data significantly decreased the number of epoochs required for the network to learn. This paper also proved that the binary dataset is a serious limitation for the convergence of neural network, so the standardization is a must in such case where the 0’s inputs simply neglect the connections in the neural network. The data set used in this paper are features extracted from gel electrophoresis images and that open the door for using artificial intelligence in such areas.

Training accurate and compact one-stage object detection networks with heuristic knowledge

A training scheme called region-refocusing (RR) is proposed to improve the accuracy and accelerate the convergence of compact one-stage detection neural networks. Main contributions are as follows: (1) the RR mask is first proposed to incorporate the position information and the significance of objects, whereby the regions containing objects can be learned selectively by the compact student detector, which leads to more reasonable feature expressions; (2) within the RR training framework, the selected objectness features from the large teacher detector are utilized to enrich the supervision information and enhance the loss functions for training the student detector, which eventually contributes to rapid convergence and accurate detection; (3) by virtue of the RR scheme, the mean average precision (mAP) of the compact detector can be significantly improved even if the model is initialized from scratch. Superiority of RR has been verified on several benchmark data sets in comparison with other training schemes; the mAP of the well-known tiny-YOLOv2 can be improved from 57.4% to 63.8% by 6.4 points on the VOC2007 test set when the weights are pretrained on ImageNet. Remarkably, when the pretraining process is omitted, it yields a significant boost of mAP by 22.6 points compared with plain training scheme, which demonstrates the robustness and high efficiency of the RR training scheme. Meanwhile, the compact one-stage detector trained with our framework is competent to be deployed on resource-constrained devices for the competitive precision as well as having a lower requirement for computing power.

Global exponential convergence for impulsive inertial complex-valued neural networks with time-varying delays

This paper focuses on the exponential convergence of impulsive inertial complex-valued neural networks with time-varying delays. The system can be expressed as a first order differential equation by selecting a proper variable substitution. By constructing proper Lyapunov–Krasovskii functionals and using inequality techniques, some delay-dependent sufficient conditions in linear matrix inequality form are proposed to ascertain the global exponential convergence of the addressed neural networks with two classes of complex-valued activation functions. The framework of the exponential convergence ball domain in which all trajectories converge is also given. Meanwhile, the obtained results here do not meet that the derivatives of the time-varying delays are less than one and there are also no limit to the strength of impulses. The methods here can also be applied to deal with multistable and monostable neural networks because of making no hypotheses on the amount of the equilibrium points. Finally, two examples are given to demonstrate the validity of the theoretical results.

Global exponential convergence of neutral-type competitive neural networks with multi-proportional delays, distributed delays and time-varying delay in leakage delays

This paper is concerned with a class of neutral-type competitive neural networks with multi-proportional delays, distributed delays and leakage delays. By employing the differential inequality theory, some sufficient conditions are given to ensure that all solutions of the addressed system converge exponentially to zero vector. An illustrative example is also given at the end of this paper to show the effectiveness of our results.

Neural Network Training With Levenberg-Marquardt and Adaptable Weight Compression.

Difficult experiments in training neural networks often fail to converge due to what is known as the flat-spot problem, where the gradient of hidden neurons in the network diminishes in value, rending the weight update process ineffective. Whereas a first-order algorithm can address this issue by learning parameters to normalize neuron activations, the second-order algorithms cannot afford additional parameters given that they include a large Jacobian matrix calculation. This paper proposes Levenberg-Marquardt with weight compression (LM-WC), which combats the flat-spot problem by compressing neuron weights to push neuron activation out of the saturated region and close to the linear region. The presented algorithm requires no additional learned parameters and contains an adaptable compression parameter, which is adjusted to avoid training failure and increase the probability of neural network convergence. Several experiments are presented and discussed to demonstrate the success of LM-WC against standard LM and LM with random restarts on benchmark data sets for varying network architectures. Our results suggest that the LM-WC algorithm can improve training success by 10 times or more compared with other methods.

Convergence of Neutral Type Fuzzy Cellular Neural Networks with D Operator

A model of neutral type fuzzy cellular neural networks with D operator is proposed. Applying differential inequality techniques, several sufficient conditions are derived to ensure the global exponential convergence of solutions for the proposed neural networks. Finally, a numerical simulation example is given to illustrate the effectiveness of the obtained results.

A New Method for Detecting P300 Signals by Using Deep Learning: Hyperparameter Tuning in High-Dimensional Space by Minimizing Nonconvex Error Function.

Background:P300 signal detection is an essential problem in many fields of Brain-Computer Interface (BCI) systems. Although deep neural networks have almost ubiquitously used in P300 detection, in such networks, increasing the number of dimensions leads to growth ratio of saddle points to local minimums. This phenomenon results in slow convergence in deep neural network. Hyperparameter tuning is one of the approaches in deep learning, which leads to fast convergence because of its ability to find better local minimums. In this paper, a new adaptive hyperparameter tuning method is proposed to improve training of Convolutional Neural Networks (CNNs).Methods:The aim of this paper is to introduce a novel method to improve the performance of deep neural networks in P300 signal detection. To reach this purpose, the proposed method transferred the non-convex error function of CNN) into Lagranging paradigm, then, Newton and dual active set techniques are utilized for hyperparameter tuning in order to minimize error of objective function in high dimensional space of CNN.Results:The proposed method was implemented on MATLAB 2017 package and its performance was evaluated on dataset of Ecole Polytechnique Fédérale de Lausanne (EPFL) BCI group. The obtained results depicted that the proposed method detected the P300 signals with 95.34% classification accuracy in parallel with high True Positive Rate (i.e., 92.9%) and low False Positive Rate (i.e., 0.77%).Conclusions:To estimate the performance of the proposed method, the achieved results were compared with the results of Naive Hyperparameter (NHP) tuning method. The comparisons depicted the superiority of the proposed method against its alternative, in such way that the best accuracy by using the proposed method was 6.44%, better than the accuracy of the alternative method.

Research on convolutional neural network based on improved Relu piecewise activation function

With the continuous development of deep learning, convolution neural network with its excellent recognition performance obtains a series of major breakthrough results in target detection, image recognition and other fields. An improved ReLu segmentation correction Activate function is proposed, by improving the traditional convolution neural network, adding the local response normalization layer, and using the maximum stacking and so on. Based on the Google depth learning platform TensorFlow, the activation function is used to construct the modified convolution neural network structure model, using the CIFAR-10 data set as the neural network input for the model training and evaluation. We analyze effects of different neuron activation function on the neural network convergence speed and the accuracy of image recognition. The experimental results show that using the improved unsaturated nonlinear segment activation function SignReLu, the convergence rate is faster, the gradient vanishing problem is effectively alleviated, and the accuracy of neural network identification is improved obviously.