Abstract
We present a general approach with reinforcement learning (RL) to approximate dynamic oracles for transition systems where exact dynamic oracles are difficult to derive. We treat oracle parsing as a reinforcement learning problem, design the reward function inspired by the classical dynamic oracle, and use Deep Q-Learning (DQN) techniques to train the oracle with gold trees as features. The combination of a priori knowledge and data-driven methods enables an efficient dynamic oracle, which improves the parser performance over static oracles in several transition systems.
Highlights
Greedy transition-based dependency parsers trained with static oracles are very efficient but suffer from the error propagation problem. Goldberg and Nivre (2012, 2013) laid the foundation of dynamic oracles to train the parser with imitation learning methods to alleviate the problem
Our work provides an initial attempt to combine the advantages of reinforcement learning and imitation learning for structured prediction in the case of dependency parsing
We compare the performance of the parser trained by the Approximate Dynamic Oracle (ADO) against the static oracle or the exact dynamic oracle (EDO) if available
Summary
Greedy transition-based dependency parsers trained with static oracles are very efficient but suffer from the error propagation problem. Goldberg and Nivre (2012, 2013) laid the foundation of dynamic oracles to train the parser with imitation learning methods to alleviate the problem. Le and Fokkens (2017) took the reinforcement learning approach (Maes et al, 2009) by directly optimizing the parser towards the reward (i.e., the correct arcs) instead of the the correct action, no oracle is required. Both approaches circumvent the difficulty in designing the oracle cost function by using the parser to (1) explore the cost of each action, and (2) explore erroneous states to alleviate error propagation
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have