Variable-Length Chromosome for Optimizing the Structure of Recurrent Neural Network

Abstract

Designing a structure of Recurrent Neural Network (RNN) can be performed either manually or automatically using a Neural Architecture Search (NAS). Automatically approach is more preferred because manually approach to designing an RNN structure can be a time consuming and error-prone process, whereas NAS uses optimization algorithms such as Evolutionary Algorithm (EA) to find the optimal structure. As one of the NAS methods, a Neuro Evolution of Augmenting Topology (NEAT) searches a neural network structure constructively by adding a new neuron through mutation, which is time-consuming for the RNN structures with large neurons. Genetic Algorithm (GA) is one of the EA that is commonly used to solve optimization problems. Fix length chromosome representation dominates GA filed, this representation suitable for fix length solution. Finding the optimal structure of RNN, the number of nodes and its connections cannot be determined. In this research, a Variable-Length Chromosome Genetic Algorithm (VLC-GA) is exploited to represent RNN structure with a different number of nodes. VLC-GA able to evolve the structure of RNN constructively and destructively. Therefore, the processing time for a larger structure can be reduced. Evaluation for a language modeling task shows that it is capable of reaching a lower perplexity than NAS with Reinforcement Learning (RL).