Implements of Transformer in NLP and DKT

Abstract

Transformer is a strong model proposed by Google team in 2017. It was a huge improvement that it entirely abandons the mechanism of Recurrent Neural Network (RNN) and Convolutional Neural Network (RNN). As a result, soon it became a popular choice in a diversity of scenarios. A typical implement of Transformer is for handling text-like input sequences, such as Natural Language Process (NLP) and Knowledge Tracing (KT). Although Transformer is a strong model, it still has a number of improvements. Some state-of-the-art Deep-Learning-based models (e.g., BERT in [2], SAINT in [3], etc.) are based on Transformer. In this paper, I give some examples of application of Transformer or Transformer-based models and summarize the pros and cons of Transformer.