Q-Learning for Shift-Reduce Parsing in Indonesian Tree-LSTM-Based Text Generation

Abstract

Tree-LSTM algorithm accommodates tree structure processing to extract information outside the linear sequence pattern. The use of Tree-LSTM in text generation problems requires the help of an external parser at each generation iteration. Developing a good parser demands the representation of complex features and relies heavily on the grammar of the corpus. The limited corpus results in an insufficient number of vocabs for a grammar-based parser, making it less natural to link the text generation process. This research aims to solve the problem of limited corpus by proposing the use of a Reinforcement Learning algorithm in the formation of constituency trees, which link the sentence generation process given a seed phrase as the input in the Tree-LSTM model. The tree production process is modeled as a Markov’s decision process, where a set of states consists of word embedding vectors, and a set of actions of {Shift, Reduce}. The Deep Q-Network model as an approximator of the Q-Learning algorithm is trained to obtain optimal weights in representing the Q-value function. The test results on perplexity-based evaluation show that the proposed Tree-LSTM and Q-Learning combination model achieves values 9.60 and 4.60 for two kinds of corpus with 205 and 1,000 sentences, respectively, better than the Shift-All model. Human evaluation of Friedman test and posthoc analysis showed that all five respondents tended to give the same assessment for the combination model of Tree-LSTM and Q-Learning, which on average outperforms two other nongrammar models, i.e., Shift-All and Reduce-All.