Implementing associative memories by Echo State Network for the applications of natural language processing

Abstract

This paper presents Echo State Network (ESN) based associative memories and their applications to English words. Among the papers describing ESN applications in general, some papers deal with natural language processing (NLP). Those NLP papers utilize an ESN’s property of temporal signal learning to translate an English sentence into its corresponding predicate logic formula. In practical NLP applications, input sentences often include misspelled or undefined words. To cope with such problems, we constructed a training algorithm to realize an ESN-based associative memory. This algorithm can be used in two ways: auto-associative one when the input and output patterns are the same in each pair of the training data, and hetero-associative one otherwise. In this paper, we firstly show the basic performance of an ESN-based auto-associative memory by applying it to two-dimensional images. We made sure that the performance is improved by averaging the result of multiple runs. Secondly, we describe the performances of ESN-based auto- and hetero-associative memories when being applied to English words. The former is to recall correct English words from incomplete spelling ones, and the latter is to chain two different English words of an input and its corresponding output. The former performance in recalling 26 incomplete words can be improved by averaging the outputs of multiple runs, and the improved success rate of correct words is around 65%. The latter performance in chaining 26 words attains the success rate of 83.4%.