Analysis of different transformer variants

Abstract

Efficient self-attention models are crucial in applications where long sequences are modeled, such as documents, images, and videos, are often composed of relatively large numbers of pixels or tokens. Therefore, the efficiency of dealing with long sequences is crucial for the broad adoption of Transformers. This paper talks about four transformer variants: Longformer, Transformer-XL, Big Bird, and Star transformer. This paper makes a comparative analysis of the four varieties theoretical methods, and obtains the advantages of the four methods and their applicable fields. Longformer has lower complexity and can be used for various document-level tasks. Transformer-XL improves the accuracy by addressing context fragmentation. BigBird replaces the Bert-like whole attention mechanism with Block Sparse Attention, which has achieved the SOTA effect on many tasks with long text sequences, such as long text summaries, long text questions, and answers. Star-Transformer reduces the time complexity to and performs well in the synthetic data set.