Comparison of convergent and independent pathways in neural networks during second-order conditioning and blocking procedures.

Supplemental Material for Comparison of Convergent and Independent Pathways in Neural Networks During Second-Order Conditioning and Blocking Procedures

BackgroundThe usage of Artificial Neural Networks (ANNs) for genome-enabled classifications and establishing genome-phenotype correlations have been investigated more extensively over the past few years. The reason for this is that ANNs are good approximates of complex functions, so classification can be performed without the need for explicitly defined input-output model. This engineering tool can be applied for optimization of existing methods for disease/syndrome classification. Cytogenetic and molecular analyses are the most frequent tests used in prenatal diagnostic for the early detection of Turner, Klinefelter, Patau, Edwards and Down syndrome. These procedures can be lengthy, repetitive; and often employ invasive techniques so a robust automated method for classifying and reporting prenatal diagnostics would greatly help the clinicians with their routine work.MethodsThe database consisted of data collected from 2500 pregnant woman that came to the Institute of Gynecology, Infertility and Perinatology “Mehmedbasic” for routine antenatal care between January 2000 and December 2016. During first trimester all women were subject to screening test where values of maternal serum pregnancy-associated plasma protein A (PAPP-A) and free beta human chorionic gonadotropin (β-hCG) were measured. Also, fetal nuchal translucency thickness and the presence or absence of the nasal bone was observed using ultrasound.ResultsThe architectures of linear feedforward and feedback neural networks were investigated for various training data distributions and number of neurons in hidden layer. Feedback neural network architecture out performed feedforward neural network architecture in predictive ability for all five aneuploidy prenatal syndrome classes. Feedforward neural network with 15 neurons in hidden layer achieved classification sensitivity of 92.00%. Classification sensitivity of feedback (Elman’s) neural network was 99.00%. Average accuracy of feedforward neural network was 89.6% and for feedback was 98.8%.ConclusionThe results presented in this paper prove that an expert diagnostic system based on neural networks can be efficiently used for classification of five aneuploidy syndromes, covered with this study, based on first trimester maternal serum screening data, ultrasonographic findings and patient demographics. Developed Expert System proved to be simple, robust, and powerful in properly classifying prenatal aneuploidy syndromes.

In this dissertation the problem of the training of feedforward artificial neural networks and its applications are considered. The presentation of the topics and the results are organized as follows: In the first chapter, the artificial neural networks are introduced. Initially, the benefits of the use of artificial neural networks are presented. In the sequence, the structure and their functionality are presented. More specifically, the derivation of the artificial neurons from the biological ones is presented followed by the presentation of the architecture of the feedforward neural networks. The historical notes and the use of neural networks in real world problems are concluding the first chapter. In Chapter 2, the existing training algorithms for the feedforward neural networks are considered. First, a summary of the training problem and its mathematical formulation, that corresponds to the uncostrained minimization of a cost function, are given. In the sequence, training algorithms based on the steepest descent, Newton, variable metric and conjugate gradient methods are presented. Furthermore, the weight space, the error surface and the techniques of the initialization of the weights are described. Their influence in the training procedure is discussed. In Chapter 3, a new training algorithm for feedforward neural networks based on the backpropagation algorithm and the automatic two-point step size (learning rate) is presented. The algorithm uses the steepest descent search direction while the learning rate parameter is calculated by minimizing the standard secant equation. Furthermore, a new learning rate parameter is derived by minimizing the modified secant equation introduced by Zhang, that uses both gradient and function value information. In the sequece a switching mechanism is incorporated into the algorithm so that the appropriate stepsize to be chosen according to the status of the current iterative point. Finaly, the global convergence of the proposed algorithm is studied and the results of some numerical experiments are presented. In Chapter 4, some efficient training algorithms, based on conjugate gradient optimization methods, are presented. In addition to the existing conjugate gradient training algorithms, we introduce Perry's conjugate gradient method as a training algorithm. Furthermore, a new class of conjugate gradient methods is proposed, called self-scaled conjugate gradient methods, which are derived from the principles of Hestenes-Stiefel, Fletcher-Reeves, Polak-Ribiere and Perry's method. This class is based on the spectral scaling parameter. Furthermore, we incorporate to the conjugate gradient training algorithms an efficient line search technique based on the Wolfe conditions and on safeguarded cubic interpolation. In addition, the initial learning rate parameter, fed to the line search technique, was automatically adapted at each iteration by a closed formula. Finally, an efficient restarting procedure was employed in order to further improve the effectiveness of the conjugate gradient training algorithms and prove their global convergence. Experimental results show that, in general, the new class of methods can perform better with a much lower computational cost and better success performance. In the last chapter of this dissertation, the Perry's self-scaled conjugate gradient training algorithm that was presented in the previous chapter was isolated and modified. More specifically, the main characteristics of the training algorithm were maintained but in this case a different line search strategy based on the nonmonotone Wolfe conditions was utilized. Furthermore, a new initial learning rate parameter was introduced for use in conjunction with the self-scaled conjugate gradient training algorithm that seems to be more effective from the initial learning rate parameter, proposed by Shanno, when used with the nonmonotone line search technique. In the sequence the experimental results for differrent training problems are presented. Finally, a feedforward neural network with the proposed algorithm for the problem of brain astrocytomas grading was trained and compared the results with those achieved by a probabilistic neural network. The dissertation is concluded with the Appendix A', where the training problems used for the evaluation of the proposed training algorithms are presented.

Objectives: This paper analyses the performance of single neuron cascade neural network with existing neural networks such as feed forward and radial basis function neural network for face recognition system. Methods: Face recognition system performance is based on the feature extraction and neural network architecture. Principal component analysis method is used for feature extraction and the extracted feature vectors are used to train the network. Using single neuron cascade architecture images are recognized. In the hidden layer single neuron is added one by one till the performance is achieved. Network is trained by set of train image and tested by a new test image. Recognition accuracy is calculated based on the recognized image. Findings: An effective classifier is identified for face recognition system. In this paper single neuron cascaded neural network is proposed for classification. In Feed forward Neural network the neurons in a layer get input from the previous layer and feed their output to the next layer. In Cascade neural network architecture the input to any layer includes all the outputs and the inputs from previous layers, which results in a cascaded interconnection between layers leading to more compact structures. Network design by cascading one neuron at a time until the desired performance is obtained can be automated. The proposed method gives systematic approach to design. It combines the advantages of both single layer feed forward neural network and multilayer feed forward neural network. Performance of the network is presented in terms of average recognition accuracy. Number of training image and test images are gradually increased from lower number samples per subject. If the number of training images is more recognition accuracy is improved. Proposed single neuron cascaded neural network out performs the existing network. Applications: This proposed method plays vital role in the field of pattern recognition, vision and human computer interactive based applications such as face recognition, surveillance, criminal identification and pass port verification. Keywords: Artificial Neural Network, Cascade Neural Network, Face Recognition, ORL database, Principal Component Analysis

Self Normalizing Neural Networks (SNN) proposed on Feed Forward Neural Networks (FNN) outperform regular FNN architectures in various machine learning tasks. Particularly in the domain of Computer Vision, the activation function Scaled Exponential Linear Units (SELU) proposed for SNNs, perform better than other non linear activations such as ReLU. The goal of SNN is to produce a normalized output for a normalized input. Established neural network architectures like feed forward networks and Convolutional Neural Networks (CNN) lack the intrinsic nature of normalizing outputs. Hence, requiring additional layers such as Batch Normalization. Despite the success of SNNs, their characteristic features on other network architectures like CNN haven’t been explored, especially in the domain of Natural Language Processing. In this paper we aim to show the effectiveness of proposed, Self Normalizing Convolutional Neural Networks (SCNN) on text classification. We analyze their performance with the standard CNN architecture used on several text classification datasets. Our experiments demonstrate that SCNN achieves comparable results to standard CNN model with significantly fewer parameters. Furthermore it also outperforms CNN with equal number of parameters.KeywordsSelf normalizing neural networksConvolutional neural networksText classification

Center of pressure (COP) trajectories of human can maintain regulation of forward progression and stability of lateral sway during walking. The insole pressure system can only detect COP trajectories of each foot during single stance. In this study, we developed artificial neural network models that could present COP trajectories in an integrated coordinate system during a complete gait cycle using pressure information of the insole system. A feed forward artificial neural network (FFANN) and a long short-term memory (LSTM) model were developed. For FFANN, among 198 pressure sensors from Pedar-X insoles, proper input variables were selected using sequential forward selection to reduce input dimension. The LSTM model used all 198 signals as inputs because of its self-learning characteristic. As results of cross-validation, the FFANN model showed correlation coefficients of 0.98-0.99 and 0.93-0.95 in anterior/posterior and medial/lateral directions, respectively. For the LSTM model, correlation coefficients were similar to those of FFANN. However, the relative root mean square error (12.5%) of the FFANN model was higher than that (9.8%) of the LSTM model in medial/lateral direction (p = 0.03). This study can be used for quantitative evaluation of clinical diagnosis and rehabilitation status for patient with various diseases through further training using varied databases. Graphical abstract Architectures of neural networks developed in this study (a feed forward artificial neural network; b LSTM network).

A feedforward neural network is a computing device whose processing units (the nodes) are distributed in adjacent layers connected through unidirectional links (the weights).Feedforward networks are widely used for pattern recognition. Here two feedforward networks are taken into consideration, Multi Layer Perceptron and Radial Basis Network. while designing these networks problem involves in finding the architecture which is efficient in terms of training time. In this paper different data samples will be presented to RBF and Multi Layer network and the best network selection will be done on the basis of minimum time taken by the network for training. Keywords- Feed forward network, Multi Layer Perceptron Neural Networks, Radial Basis Network, Spread. I. INTRODUCTION Pattern recognition is the study of how machines can observe the environment, learn to explore patterns of interest from their background, and make reasonable decisions about the classes of the patterns. the main properties of neural networks are that they have the ability to learn nonlinear input-output relationships, use sequential training procedures, and adapt themselves to the data. the most commonly used neural network family for pattern classification tasks is the feed-forward network, which includes Multi Layer Perceptron(MLP) and Radial-Basis Function (RBF) networks(9). Neural networks can be viewed as massively parallel computing systems consisting of an extremely large number of simple processors with many interconnections. Neural network models attempt to use some organizational principles (such as learning, generalization, adaptivity, fault tolerance and distributed representation. computation) in a network of weighted directed graphs in which the nodes are artificial neurons and directed edges (with weights) are connections between neuron outputs and neuron inputs. The main characteristics of neural networks are that they have the ability to learn complex nonlinear input-output relationships, use sequential training procedures, and adapt themselves to the data.

The basolateral complex of the amygdala (ABL) is involved in processing information about stimulus motivational value. However, it is not clear whether the ABL is critical for acquisition, maintenance, or expression of this information. Our previous work has shown that ABL lesions made prior to training, block acquisition of an appetitive Pavlovian second-order conditioning task, in which performance is thought to depend on the acquisition of motivational (conditioned reinforcement) value by the first-order conditioned stimulus (CS). The present experiments examined the effects of ABL lesions made after first-order conditioning, when the CS acquires motivational value, but before second-order conditioning, the test for acquired value of that CS. Rats received pairings of a visual CS with a food reinforcer. They then received bilateral sham or excitotoxic lesions of the ABL. After recovery, they received pairings of a second-order auditory CS with the previously conditioned visual CS. In two experiments, both sham and lesioned rats acquired normal second-order conditioned behaviours. Some of the same rats were then given another round of second-order conditioning with novel CSs. In this case, when first-order training occurred after surgery, some second-order conditioned behaviours were impaired in lesioned rats. Tests of the associative underpinnings of second-order conditioned behaviours showed that those behaviours impaired by ABL lesions were based on stimulus-response associations. The results show that although the ABL is critical for second-order conditioning, this role is limited to acquisition of information about the motivational value of the first-order CS, and it is not critical for maintenance of this information or for its use in forming second-order associations.

A neural network is a very useful tool in artificial intelligence (AI) that can also be referred to as an ANN. An artificial neural network (ANN) is a deep learning model that has a broad range of applications in real life. The combination and interrelationship of neurons and nodes with each other facilitate the transmission of information. An ANN has a feed-forward neural network. The neurons are arranged in layers, and each layer performs a particular calculation on the incoming data. Up until the output layer, which generates the network’s ultimate output, is reached, each layer’s output is transmitted as an input to the subsequent layer. A feed-forward neural network (FFNN) is a method for finding the output of expert information. In this research, we expand upon the concept of fuzzy neural network systems and introduce feed-forward double-hierarchy linguistic neural network systems (FFDHLNNS) using Yager–Dombi aggregation operators. We also discuss the desirable properties of Yager–Dombi aggregation operators. Moreover, we describe double-hierarchy linguistic term sets (DHLTSs) and discuss the score function of DHLTSs and the distance between any two double-hierarchy linguistic term elements (DHLTEs). Here, we discuss different approaches to choosing a novel water purification technique on a commercial scale, as well as some variables influencing these approaches. We apply a feed-forward double-hierarchy linguistic neural network (FFDHLNN) to select the best method for water purification. Moreover, we use the extended version of the Technique for Order Preference by Similarity to Ideal Solution (extended TOPSIS) method and the grey relational analysis (GRA) method for the verification of our suggested approach. Remarkably, both approaches yield almost the same results as those obtained using our proposed method. The proposed models were compared with other existing models of decision support systems, and the comparison demonstrated that the proposed models are feasible and valid decision support systems. The proposed technique is more reliable and accurate for the selection of large-scale water purification methods.

The goal of this paper is to present interesting way how to model and predict nonlinear systems using recurrent neural network. This type of artificial neural networks is underestimated and marginalized. Nevertheless, it offers superior modelling features at reasonable computational costs. This contribution is focused on Elman Neural Network, two-layered recurrent neural network. The abilities of this network are presented in the nonlinear system control. The task of the controller is to control the liquid level in the second of two interconnected cylindrical tanks. The mathematical model of the realtime system was derived in order to test predictor and consequently the controller in Matlab/Simulink simulations. INTRODUCTION Model predictive control (MPC) (Camacho and Bordons 2007) is a very popular concept for the development and tuning of nonlinear controllers in the presence of input, output or state constraints. Many predictive control techniques based on MPC that use artificial neural network (ANN) as a predictor are established on multilayer feed-forward neural networks (Hagan et al. 2002), (Kanjilal 1995). In spite the multilayer feedforward neural networks (MFFNNs) have many advantages such as simple design and scalability, they have also many drawbacks such as long training times and choice of an appropriate learning stop time (the over-learning versus the early stopping problem). However, there is quite a number of types ANNs suitable for the modelling and prediction, for instance adaptive linear networks, radial basis function networks and recurrent networks (Liu 2001), (Meszaros et al. 1999), (Koker 2006). This paper is divided as follows: After short introduction to the recurrent neural networks, the used model predictive controller is explained. Then the model of the real time system is derived. After that the identification of the predictor (training of the artificial neural network) is described. When the identification is finished, the paper focuses on the model predictive control and evaluation of results. The contribution is finished by some concluding remarks. RECURRENT NEURAL NETWORKS Recurrent neural networks (sometimes are these networks called feedback neural networks) can be distinguished from feed-forward neural networks in that they have a loopback connection (Figure 1). In its most general form recurrent network consist of a set of processing units, while the output of each unit is fed as input to all other units including the same unit. With each link connecting any two units, a weight is associated which determines the amount of output a unit feeds as input to the other unit (Yegnanarayana 2005). Figure 1: Example of Recurrent Neural Network Recurrent neural networks have superior temporal and spatial behaviours, such as stable and unstable fixed points and limit cycles, and chaotic behaviours. These behaviours can be utilized to model certain cognitive functions, such as associative memory, unsupervised learning, self-organizing maps, and temporal reasoning (He 1999). Elman Neural Networks One of the most known recurrent neural networks is Elman neural network (Elman 1990). Typical Elman network has one hidden layer with delayed feedback. The Elman neural network is capable of providing the standard state-space representation for dynamic systems. This is the reason why this network architecture is utilized as a recurrent neural equalizer. Proceedings 23rd European Conference on Modelling and Simulation ©ECMS Javier Otamendi, Andrzej Bargiela, Jose Luis Montes, Luis Miguel Doncel Pedrera (Editors) ISBN: 978-0-9553018-8-9 / ISBN: 978-0-9553018-9-6 (CD) Generally, this network is considered as a special kind feed-forward network, including additional memory neurons and local feedback (Koker 2006). Typical structure of Elman neural network is depicted in fig. 2. Figure 2: Elman Neural Network MODEL PREDICTIVE CONTROL There are various approaches to predictive control by artificial neural networks. Generally we can say that these methods use ANN as the plant model in order to get its output predictions. The most used approach is model predictive control (Camacho and Bordons 1995). MPC is a broad control strategy applicable to both linear and nonlinear processes. The main idea of MPC algorithms is to use a dynamical model of process to predict the effect of future control actions on the output of the process. Hence, the controller calculates the control input that will optimize the performance criterion over a specified future time horizon:

PurposeThe purpose of this paper is to conduct a comprehensive review of the noteworthy contributions made in the area of the Feedforward neural network (FNN) to improve its generalization performance and convergence rate (learning speed); to identify new research directions that will help researchers to design new, simple and efficient algorithms and users to implement optimal designed FNNs for solving complex problems; and to explore the wide applications of the reviewed FNN algorithms in solving real-world management, engineering and health sciences problems and demonstrate the advantages of these algorithms in enhancing decision making for practical operations.Design/methodology/approachThe FNN has gained much popularity during the last three decades. Therefore, the authors have focused on algorithms proposed during the last three decades. The selected databases were searched with popular keywords: “generalization performance,” “learning rate,” “overfitting” and “fixed and cascade architecture.” Combinations of the keywords were also used to get more relevant results. Duplicated articles in the databases, non-English language, and matched keywords but out of scope, were discarded.FindingsThe authors studied a total of 80 articles and classified them into six categories according to the nature of the algorithms proposed in these articles which aimed at improving the generalization performance and convergence rate of FNNs. To review and discuss all the six categories would result in the paper being too long. Therefore, the authors further divided the six categories into two parts (i.e. Part I and Part II). The current paper, Part I, investigates two categories that focus on learning algorithms (i.e. gradient learning algorithms for network training and gradient-free learning algorithms). Furthermore, the remaining four categories which mainly explore optimization techniques are reviewed in Part II (i.e. optimization algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms, constructive topology neural networks and metaheuristic search algorithms). For the sake of simplicity, the paper entitled “Machine learning facilitated business intelligence (Part II): Neural networks optimization techniques and applications” is referred to as Part II. This results in a division of 80 articles into 38 and 42 for Part I and Part II, respectively. After discussing the FNN algorithms with their technical merits and limitations, along with real-world management, engineering and health sciences applications for each individual category, the authors suggest seven (three in Part I and other four in Part II) new future directions which can contribute to strengthening the literature.Research limitations/implicationsThe FNN contributions are numerous and cannot be covered in a single study. The authors remain focused on learning algorithms and optimization techniques, along with their application to real-world problems, proposing to improve the generalization performance and convergence rate of FNNs with the characteristics of computing optimal hyperparameters, connection weights, hidden units, selecting an appropriate network architecture rather than trial and error approaches and avoiding overfitting.Practical implicationsThis study will help researchers and practitioners to deeply understand the existing algorithms merits of FNNs with limitations, research gaps, application areas and changes in research studies in the last three decades. Moreover, the user, after having in-depth knowledge by understanding the applications of algorithms in the real world, may apply appropriate FNN algorithms to get optimal results in the shortest possible time, with less effort, for their specific application area problems.Originality/valueThe existing literature surveys are limited in scope due to comparative study of the algorithms, studying algorithms application areas and focusing on specific techniques. This implies that the existing surveys are focused on studying some specific algorithms or their applications (e.g. pruning algorithms, constructive algorithms, etc.). In this work, the authors propose a comprehensive review of different categories, along with their real-world applications, that may affect FNN generalization performance and convergence rate. This makes the classification scheme novel and significant.

Existing neural network algorithms have the problems of slow convergence and low accuracy. In response to this phenomenon, this paper presents a neural network blind equalization algorithm based on feed-forward neural network. And we proposed feed-forward neural network blind equalization algorithm by research of traditional neural network blind equalization algorithm. And it is using a feed-forward neural network of the hidden layer to approximate the objective function. At last, we by combining the cost functions of feed- forward network to correct the acquired information. Experimental results show that the experimental results basically consistent with the expected results. By comparison with other algorithms, this algorithm has better convergence and accuracy.

Single-hidden-layer feed-forward quantum neural network based on Grover learning

The ability of a neural network to represent an input-output mapping is usually only measured in terms of the data fit according to some error criteria. This ‘black box’ approach provides little understanding of the network representation or how it should be structured. This paper investigates the topological structure of multilayer feedforward neural networks (MFNN) and explores the relationship between the numbers of neurons in the hidden layers and finite dimensional topological spaces. It is shown that a class of three layer (two hidden layer) neural networks is equivalent to a canonical form approximation of nonlinearity. This theoretical framework leads to insights about the architecture of multilayer feedforward neural networks, confirms the common belief that three layer (two hidden layer) feedforward networks are sufficient for general application and yields an approach for determining the appropriate numbers of neurons in each hidden layer.

The core of this book are deep learning methods and neural networks. This chapter considers deep feed-forward neural (FN) networks. We introduce the generic architecture of deep FN networks, and we discuss universality theorems of FN networks. We present network fitting, back-propagation, embedding layers for categorical variables and insurance-specific issues such as the balance property in network fitting, as well as network ensembling to reduce model uncertainty. This chapter is complemented by many examples on non-life insurance pricing, but also on mortality modeling, as well as tools that help to explain deep FN network regression results.