Chapter 18 - Neural Networks and Deep Learning

Abstract

This chapter deals with neural networks, starting from the early days of the perceptron and perceptron rule, and then moves on to the multilayer feed-forward neural networks and their training via the backpropagation algorithmic concept. A number of algorithms are discussed, starting from the basic gradient descent scheme up to very recently proposed popular variants. The palette of possible nonlinearities, including the ReLU, is presented and the effects of the choice of the nonlinearity on the convergence of the training algorithm are discussed, also in relation to the cost function that is selected. Regularization techniques, including the dropout method, are presented. In the sequel, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are introduced in some depth, alongside some very popular deep networks that are currently in use. In the sequel, generative networks are presented, starting from the more classical models based on Boltzmann machines and deep belief networks and moving on to the more recent advances including variational autoencoders and generative adversarial networks (GANs). Finally, capsule networks are introduced and discussed. A case study from the field of natural language processing (NLP) completes the chapter.