Researches Advanced in the Development and Application of Transformers

Abstract

The basic task of feature learning is to use algorithms to allow machines to automatically learn useful data and its features during the model building process. The quality of the learned features will greatly affect the results of downstream tasks. Early feature learning methods relied on handcrafted features. Thanks to the development of deep learning, feature learning methods based on convolutional neural networks have greatly improved the quality of features. However, with the increasing scale of training data and the increasing complexity of modeling tasks, deep neural network Transformer based on self-attention mechanism and parallel data processing has gradually become a new research hotspot. Transformer can adaptively and selectively select contextual information and key semantic information in a scene by covering attention networks and fully connected layers and has become an important research area for computer vision and natural language processing tasks. This paper reviews the basic principles and development of Transformer, focuses on its application in CV and NLP, and analyzes effective Transformer-based models. Finally, the challenges faced, and future trends of transformer models are summarized.