Formula omitted]-Scaled-attention: A novel fast attention mechanism for efficient modeling of protein sequences

Abstract

Attention-based deep networks have been successfully applied on textual data in the field of NLP. However, their application on protein sequences poses unexplored challenges because, unlike the plain text words, protein words have weak semantics. Such challenges include: (i) vanishing attention score problem and (ii) high variations in the attention distribution. Here, we introduce a novel λ-scaled attention technique for a fast and efficient modeling of the protein sequences aimed at addressing these two problems. This is then used to develop the λ-scaled attention network and is evaluated for the task of protein function prediction implemented at the protein sub-sequence level. Experiments on the datasets for biological process (BP) and molecular function (MF) showed significant improvements in the F1 score values for the proposed λ-scaled attention technique over its counterpart approach based on the standard attention technique (+1.68% for BP and +5.27% for MF) and state-of-the-art multi-segment based ProtVecGen-Plus approach (+4.70% for BP and +5.30% for MF). Further, fast convergence (converging in half the number of epochs) and efficient learning (captured by low difference between the training and validation losses) were also observed during the training process.