An Efficient Transformer with Distance-aware Attention

Abstract

In recent years, the transformer model has become one of the main highlights of advances in natural language processing (NLP). The attention mechanism of the transformer model makes it possible to track the relations between words across very long text sequences in both forward and reverse directions. However, the complexity of the attention mechanism is quadratic and introduces a performance bottleneck in the transformer. We propose a distance-aware attention mechanism which integrates the locality information by assigning different weights to query-key pairs according to the distance between the query and the key. By doing so, we in fact shrink the dimension of the matrix in the vector matrix multiplication and reduce the complexity of the attention to O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /m). Experiments show that the distance-aware attention is superior to or close to the original model and other variants in various NLP tasks.