An Entropy Metric for Regular Grammar Classification and Learning with Recurrent Neural Networks.

Kaixuan Zhang,C Lee Giles,Qinglong Wang

doi:10.3390/e23010127

Abstract

Recently, there has been a resurgence of formal language theory in deep learning research. However, most research focused on the more practical problems of attempting to represent symbolic knowledge by machine learning. In contrast, there has been limited research on exploring the fundamental connection between them. To obtain a better understanding of the internal structures of regular grammars and their corresponding complexity, we focus on categorizing regular grammars by using both theoretical analysis and empirical evidence. Specifically, motivated by the concentric ring representation, we relaxed the original order information and introduced an entropy metric for describing the complexity of different regular grammars. Based on the entropy metric, we categorized regular grammars into three disjoint subclasses: the polynomial, exponential and proportional classes. In addition, several classification theorems are provided for different representations of regular grammars. Our analysis was validated by examining the process of learning grammars with multiple recurrent neural networks. Our results show that as expected more complex grammars are generally more difficult to learn.

Highlights

Regular grammars (RGs) have been widely studied in the theory of computation and intensively applied in natural language processing, compiler construction, software design, parsing and formal verification [1,2,3,4]
Our empirical results show that the categorization of RGs is related to the difficulty of recurrent neural networks (RNNs) to learn these grammars, and the implementation is publicly available
To measure the complexity of regular grammar, we introduced an entropy metric based on the concentric ring representation, which essentially reflects the difficulty in training RNNs to learn the grammar

Summary

Introduction

Regular grammars (RGs) have been widely studied in the theory of computation and intensively applied in natural language processing, compiler construction, software design, parsing and formal verification [1,2,3,4]. Despite their importance and pervasiveness, there is limited research [5,6,7] investigating the internal structure of regular grammars. Recent work has shown that certain types of regular grammars can be more learned by recurrent networks [9,10] This is important in that it provides crucial insights in understanding regular grammar complexity. Understanding the learning process of regular grammar help differentiating different recurrent models [11,12,13]

Objectives

Results

Conclusion