Connection Weights Of Network Research Articles

Exponential Discretization of Weights of Neural Network Connections in Pre-Trained Neural Network. Part II: Correlation Maximization

In this article, we develop method of linear and exponential quantization of neural network weights. We improve it by means of maximizing correlations between the initial and quantized weights taking into account the weight density distribution in each layer. We perform the quantization after the neural network training without a subsequent post-training and compare our algorithm with linear and exponential quantization. The quality of the neural network VGG-16 is already satisfactory (top5 accuracy 76%) in the case of 3-bit exponential quantization. The ResNet50 and Xception neural networks show top5 accuracy at 4 bits 79% and 61%, respectively.

Analysis and Forecast of Urban Economic Vitality in Northeast China

Taking into account the passage of time, the original economic vitality index will vary with changes in social development, we use the BP neural network nearly a decade as the original GDP data for the next 30 years the GDP forecast. BP neural network in 1985, proposed by Rumelhart, the algorithm solves the system of learning problems multilayer neural network connection weights hidden layer [1].It consists of an input layer, a hidden layer, and an output layer. The principle is to continuously adjust the network weights and thresholds by transmitting errors backward and then correcting the errors to achieve the desired input-output mapping.

Inverse calculation of demolition robot based on gravitational search algorithm and differential evolution neural network

For the inverse calculation of laser-guided demolition robot, its global nonlinear mapping model from laser measuring point to joint cylinder stroke has been set up with an artificial neural network. Due to the contradiction between population diversity and convergence rate in the optimization of complex neural networks by using differential evolution, a gravitational search algorithm and differential evolution is proposed to accelerate the convergence rate of differential evolution population driven by gravity. Gravitational search algorithm and differential evolution is applied to optimize the inverse calculation neural network mapping model of demolition robot, and the algorithm simulation shows that gravity can effectively regulate the convergence process of differential evolution population. Compared with the standard differential evolution, the convergence speed and accuracy of gravitational search algorithm and differential evolution are significantly improved, which has better optimization stability. The calculation results show that the output accuracy of this gravitational and differential evolution neural network can meet the calculation requirements of the positioning control of demolition robot’s manipulator. The optimization using gravitational search algorithm and differential evolution is done with the connection weights of a neural network in this article, and as similar techniques can be applied to the other hyperparameter optimization problem. Moreover, such an inverse calculation method can provide a reference for the autonomous positioning of large hydraulic series manipulator, so as to improve the robotization level of construction machinery.

Neural Computing Enhanced Parameter Estimation for Multi-Input and Multi-Output Total Non-Linear Dynamic Models

In this paper, a gradient descent algorithm is proposed for the parameter estimation of multi-input and multi-output (MIMO) total non-linear dynamic models. Firstly, the MIMO total non-linear model is mapped to a non-completely connected feedforward neural network, that is, the parameters of the total non-linear model are mapped to the connection weights of the neural network. Then, based on the minimization of network error, a weight-updating algorithm, that is, an estimation algorithm of model parameters, is proposed with the convergence conditions of a non-completely connected feedforward network. In further determining the variables of the model set, a method of model structure detection is proposed for selecting a group of important items from the whole variable candidate set. In order to verify the usefulness of the parameter identification process, we provide a virtual bench test example for the numerical analysis and user-friendly instructions for potential applications.

Artificial Neural Networks Solve Musical Problems With Fourier Phase Spaces

How does the brain represent musical properties? Even with our growing understanding of the cognitive neuroscience of music, the answer to this question remains unclear. One method for conceiving possible representations is to use artificial neural networks, which can provide biologically plausible models of cognition. One could train networks to solve musical problems, and then study how these networks encode musical properties. However, researchers rarely examine network structure in detail because networks are difficult to interpret, and because many assume that networks capture informal or subsymbolic properties. Here we report very high correlations between network connection weights and discrete Fourier phase spaces used to represent musical sets. This is remarkable because there is no clear mathematical relationship between network learning rules and discrete Fourier analysis. That networks discover Fourier phase spaces indicates that these spaces have an important role to play outside of formal music theory. Finding phase spaces in networks raises the strong possibility that Fourier components are possible codes for musical cognition.

Design of highly effective multilayer feedforward neural network by using genetic algorithm

AbstractThis paper presents a highly effective and precise neural network method for choosing the activation functions (AFs) and tuning the learning parameters (LPs) of a multilayer feedforward neural network by using a genetic algorithm (GA). The performance of the neural network mainly depends on the learning algorithms and the network structure. The backpropagation learning algorithm is used for tuning the network connection weights, and the LPs are obtained by the GA to provide both fast and reliable learning. Also, the AFs of each neuron in the network are automatically chosen by a GA. The present study consists of 10 different functions to accomplish a better convergence of the desired input–output mapping. Test studies are performed to solve a set of two‐dimensional regression problems for the proposed genetic‐based neural network (GNN) and conventional neural network having sigmoid AFs and constant learning parameters. The proposed GNN has also been tested by applying it to three real problems in the fields of environment, medicine, and economics. Obtained results prove that the proposed GNN is more effective and reliable when compared with the classical neural network structure.

Achievements analysis of mooc English course based on fuzzy statistics and neural network clustering

At present, the field of natural language will also introduce in-depth learning, using the concept of word vector, so that the neural network can also complete the work in the field of statistics. It can be said that the neural network has begun to show its advantages in the field of natural language processing. In this paper, the author analyzes the multimedia English course based on fuzzy statistics and neural network clustering. Different factors were classified, and scores were classified according to the number of characteristics of different categories. It can be seen that with the popularization of the Internet, MOOC teaching meets the requirements of the current college English curriculum, is a breakthrough in the traditional teaching mode, improves students’ participation, and enables students to learn independently. It not only conforms to the characteristics of College students, but also improves their learning effect. In the automatic scoring stage, the quantitative text features are extracted by the feature extractor in the pre-processing stage, and then the weights of network connections obtained in the training stage are used to score the weights comprehensively. This model can better reflect students’ autonomous learning ability and language application ability.

Particle Swarm Optimization to Obtain Weights in Neural Network

This research discusses the use of a class of heuristic optimization to obtain the weights in neural network model for time series prediction. In this case, Feed Forward Neural Network (FFNN) was chosen as the class of network architecture. The heuristic algorithm determined to obtain the weights in network was Particle Swarm Optimization (PSO). It is a non-gradient optimization technique. This method was used for optimizing the connection weights of network. The lags used as the input were selected based on the strong relationship with the current. The eight architectures were conducted to improve the accuracy of the neural network model. In each architecture, we repeated the running thirty times to get the statistics of mean and variance. The comparison of the performance of various architectures based on the minimum MSE and the stability of the results is presented in this paper. The optimal number of neurons in hidden layer was determined by these criteria. The proposed procedure was applied in air pollution data, i.e. Solid Particulate Matter (SPM). The results showed that the proposed procedure gave promising results in terms of prediction accuracy. A few neurons in hidden layer are strongly recommended in choosing the optimal architecture.

Exponential Discretization of Weights of Neural Network Connections in Pre-Trained Neural Networks

To reduce random access memory (RAM) requirements and to increase speed of recognition algorithms we consider a weight discretization problem for trained neural networks. We show that an exponential discretization is preferable to a linear discretization since it allows one to achieve the same accuracy when the number of bits is 1 or 2 less. The quality of the neural network VGG-16 is already satisfactory (top5 accuracy 69%) in the case of 3 bit exponential discretization. The ResNet50 neural network shows top5 accuracy 84% at 4 bits. Other neural networks perform fairly well at 5 bits (top5 accuracies of Xception, Inception-v3, and MobileNet-v2 top5 were 87%, 90%, and 77%, respectively). At less number of bits, the accuracy decreases rapidly.

The Study of a Sales Forecast Model Based on SA-LSTM

Sales forecast is an indispensable link in the business activities of enterprises, and the accuracy of prediction is directly related to the effectiveness of enterprise sales and operation activities. In order to improve the prediction accuracy, a sales forecasting model based on LSTM is proposed. The model uses SA to optimize the initial connection weights of LSTM neural network, which solves the problem that the LSTM neural network converges to the local optimal, thus improving the network performance, and then makes an empirical analysis of the construction of the sales forecasting model based on SA-LSTM. The results show that the sales forecasting model improves the prediction accuracy, also reduces the number of iterations, and obtains a good prediction effect.

Nonlinear inversion for magnetotelluric sounding based on deep belief network

To improve magnetotelluric (MT) nonlinear inversion accuracy and stabilitythis work introduces the deep belief network (DBN) algorithm. Firstlya network frame is set up for training in different 2D MT models. The network inputs are the apparent resistivities of known modelsand the outputs are the model parameters. The optimal network structure is achieved by determining the numbers of hidden layers and network nodes. Secondlythe learning process of the DBN is implemented to obtain the optimal solution of network connection weights for known geoelectric models. Finallythe trained DBN is verified through inversion testsin which the network inputs are the apparent resistivities of unknown modelsand the outputs are the corresponding model parameters. The experiment results show that the DBN can make full use of the global searching capability of the restricted Boltzmann machine (RBM) unsupervised learning and the local optimization of the back propagation (BP) neural network supervised learning. Comparing to the traditional neural network inversionthe calculation accuracy and stability of the DBN for MT data inversion are improved significantly. And the tests on synthetic data reveal that this method can be applied to MT data inversion and achieve good results compared with the least-square regularization inversion.

Machine learning methods for better water quality prediction

In any aquatic system analysis, the modelling water quality parameters are of considerable significance. The traditional modelling methodologies are dependent on datasets that involve large amount of unknown or unspecified input data and generally consist of time-consuming processes. The implementation of artificial intelligence (AI) leads to a flexible mathematical structure that has the capability to identify non-linear and complex relationships between input and output data. There has been a major degradation of the Johor River Basin because of several developmental and human activities. Therefore, setting up of a water quality prediction model for better water resource management is of critical importance and will serve as a powerful tool. The different modelling approaches that have been implemented include: Adaptive Neuro-Fuzzy Inference System (ANFIS), Radial Basis Function Neural Networks (RBF-ANN), and Multi-Layer Perceptron Neural Networks (MLP-ANN). However, data obtained from monitoring stations and experiments are possibly polluted by noise signals as a result of random and systematic errors. Due to the presence of noise in the data, it is relatively difficult to make an accurate prediction. Hence, a Neuro-Fuzzy Inference System (WDT-ANFIS) based augmented wavelet de-noising technique has been recommended that depends on historical data of the water quality parameter. In the domain of interests, the water quality parameters primarily include ammoniacal nitrogen (AN), suspended solid (SS) and pH. In order to evaluate the impacts on the model, three evaluation techniques or assessment processes have been used. The first assessment process is dependent on the partitioning of the neural network connection weights that ascertains the significance of every input parameter in the network. On the other hand, the second and third assessment processes ascertain the most effectual input that has the potential to construct the models using a single and a combination of parameters, respectively. During these processes, two scenarios were introduced: Scenario 1 and Scenario 2. Scenario 1 constructs a prediction model for water quality parameters at every station, while Scenario 2 develops a prediction model on the basis of the value of the same parameter at the previous station (upstream). Both the scenarios are based on the value of the twelve input parameters. The field data from 2009 to 2010 was used to validate WDT-ANFIS. The WDT-ANFIS model exhibited a significant improvement in predicting accuracy for all the water quality parameters and outperformed all the recommended models. Also, the performance of Scenario 2 was observed to be more adequate than Scenario 1, with substantial improvement in the range of 0.5% to 5% for all the water quality parameters at all stations. On validating the recommended model, it was found that the model satisfactorily predicted all the water quality parameters (R2 values equal or bigger than 0.9).

Evolving Deep Convolutional Neural Networks for Image Classification

Evolutionary paradigms have been successfully applied to neural network designs for two decades. Unfortunately, these methods cannot scale well to the modern deep neural networks due to the complicated architectures and large quantities of connection weights. In this paper, we propose a new method using genetic algorithms for evolving the architectures and connection weight initialization values of a deep convolutional neural network to address image classification problems. In the proposed algorithm, an efficient variable-length gene encoding strategy is designed to represent the different building blocks and the potentially optimal depth in convolutional neural networks. In addition, a new representation scheme is developed for effectively initializing connection weights of deep convolutional neural networks, which is expected to avoid networks getting stuck into local minimum that is typically a major issue in the backward gradient-based optimization. Furthermore, a novel fitness evaluation method is proposed to speed up the heuristic search with substantially less computational resource. The proposed algorithm is examined and compared with 22 existing algorithms on nine widely used image classification tasks, including the state-of-the-art methods. The experimental results demonstrate the remarkable superiority of the proposed algorithm over the state-of-the-art designs in terms of classification error rate and the number of parameters (weights).

Evolving feedforward artificial neural networks using a two-stage approach

This paper presents a two-stage approach, denoted as CBRDE-LM, to evolve the architecture and weights of feedforward artificial neural networks. In the first stage, a collaborative binary-real differential evolution (CBRDE) is used to optimize simultaneously network architecture and connection weights of an ANN by a specific individual representation and evolutionary scheme, in which the structure is indirectly represented as binary coding and connection weights are directly encoded by real-valued coding. In the second stage, based on the resulting architecture and weights of an ANN, Levenberg-Marquardt (LM) backpropagation algorithm is adopted for fine-tuning ANN weights. The performance of the two-stage approach has been evaluated on several benchmarks. The results demonstrate that the two-stage approach can fast produce compact ANNs with good generalization ability at low computational cost.

The weights initialization methodology of unsupervised neural networks to improve clustering stability

A study on initialization of connection weights of neural networks is expected to be needed because various deep neural networks based on deep learning have attracted much attention recently. However, studies on the relation between the output value of the active function and the learning performance of the neural network with respect to the connection weight value have been conducted mainly on the supervised learning model. This paper focused on improving the efficiency of autonomous neural network model by studying the connection weight initialization as the neural network model of supervised learning. Adaptive resonance theory (ART) is a major model of autonomous neural network that tries to solve the stability–plasticity dilemma by using bottom-up weights and top-down weights. The conventional weights initialization method of ART was to uniformly set all weights, but the proposed method is to initialize by using pre-trained weights. Experiments show that the ART, which initializes the connectivity weights through the proposed method, performs clustering more reliably.

Non-iterative Knowledge Fusion in Deep Convolutional Neural Networks

Incorporation of a new knowledge into neural networks with simultaneous preservation of the previous one is known to be a nontrivial problem. This problem becomes even more complex when new knowledge is contained not in new training examples, but inside the parameters (connection weights) of another neural network. Here we propose and test two methods allowing combining the knowledge contained in separate networks. One method is based on a simple operation of summation of weights of constituent neural networks. Another method assumes incorporation of a new knowledge by modification of weights nonessential for the preservation of already stored information. We show that with these methods the knowledge from one network can be transferred into another one non-iteratively without requiring training sessions. The fused network operates efficiently, performing classification far better than a chance level. The efficiency of the methods is quantified on several publicly available data sets in classification tasks both for shallow and deep neural networks.

Evolving neural networks to follow trajectories of arbitrary complexity

Many experiments have been performed that use evolutionary algorithms for learning the topology and connection weights of a neural network that controls a robot or virtual agent. These experiments are not only performed to better understand basic biological principles, but also with the hope that with further progress of the methods, they will become competitive for automatically creating robot behaviors of interest. However, current methods are limited with respect to the (Kolmogorov) complexity of evolved behavior. Using the evolution of robot trajectories as an example, we show that by adding four features, namely (1) freezing of previously evolved structure, (2) temporal scaffolding, (3) a homogeneous transfer function for output nodes, and (4) mutations that create new pathways to outputs, to standard methods for the evolution of neural networks, we can achieve an approximately linear growth of the complexity of behavior over thousands of generations. Overall, evolved complexity is up to two orders of magnitude over that achieved by standard methods in the experiments reported here, with the major limiting factor for further growth being the available run time. Thus, the set of methods proposed here promises to be a useful addition to various current neuroevolution methods.

Single index fuzzy neural networks using locally weighted polynomial regression

The novel single index fuzzy neural network models are proposed in this paper for general machine learning problems. The proposed models are different from the usual fuzzy neural network models in that the output nodes of the networks are replaced by (nonparametric) single index models. Specifically, instead of pre-specifying the output activation functions as in the usual models, they are re-estimated adaptively during the training process via “Loess” (“LOcal regrESSion”), a powerful (nonparametric) scatterplot smoother. These estimated activation functions are not necessarily the usual sigmoidal or identity functions. It is interesting to find that in many cases the estimated output activation functions are well approximated by simple polynomial or generalized hyperbolic tangent functions. These problem-tailored simple functions can, if necessary, then be used as the actual output activation functions for neural network training and prediction. Particle swarm optimization, a commonly used evolutionary computation technique, is adopted in this study to search the optimal connection weights of the neural networks. The main advantages of the single index fuzzy neural network models are that they are well suited in situations when one lacks the information about the probability distribution of the response and it is not necessary to specify the output activation functions of the neural networks. Simulation results show that the proposed models usually provide better fits than the usual models for the data at hand.

Blind Estimation of the PN Sequence of A DSSS Signal Using A Modified Online Unsupervised Learning Machine

Direct sequence spread spectrum (DSSS) signals are now widely used in air and underwater acoustic communications. A receiver which does not know the pseudo-random (PN) sequence cannot demodulate the DSSS signal. In this paper, firstly, the principle of principal component analysis (PCA) for PN sequence estimation of the DSSS signal is analyzed, then a modified online unsupervised learning machine (LEAP) is introduced for PCA. Compared with the original LEAP, the modified LEAP has the following improvements: (1) By normalizing the system state transition matrices, the modified LEAP can obtain better robustness when the training errors occur; (2) with using variable learning steps instead of a fixed one, the modified LEAP not only converges faster but also has excellent estimation performance. When the modified LEAP is converging, we can utilize the network connection weights which are the eigenvectors of the autocorrelation matrix of the DSSS signal to estimate the PN sequence. Due to the phase ambiguity of the eigenvectors, a novel approach which is based on the properties of the PN sequence is proposed here to exclude the wrong estimated PN sequences. Simulation results showed that the methods mentioned above can estimate the PN sequence rapidly and robustly, even when the DSSS signal is far below the noise level.

An Overview of Neural Network

Neural networks represent a brain metaphor for information processing. These models are biologically inspired rather than an exact replica of how the brain actually functions. Neural networks have been shown to be very promising systems in many forecasting applications and business classification applications due to their ability to learn from the data. This article aims to provide a brief overview of artificial neural network. The artificial neural network learns by updating the network architecture and connection weights so that the network can efficiently perform a task. It can learn either from available training patterns or automatically learn from examples or input-output relations. Neural network-based models continue to achieve impressive results on longstanding machine learning problems, but establishing their capacity to reason about abstract concepts has proven difficult. Building on previous efforts to solve this important feature of general-purpose learning systems, our latest paper sets out an approach for measuring abstract reasoning in learning machines, and reveals some important insights about the nature of generalization itself. Artificial neural networks can learn by example like the way humans do. An artificial neural net is configured for a specific application like pattern recognition through a learning process. Learning in biological systems consists of adjustments to the synaptic connections that exist between neurons. This is true of artificial neural networks as well. Artificial neural networks can be applied to an increasing number of real-world problems of considerable complexity. They are used for solving problems that are too complex for conventional technologies or those types of problems that do not have an algorithmic solution.