Rethinking attention mechanism in time series classification

Abstract

The potential of attention mechanisms in time series classification (TSC) is limited owing to general drawbacks, like weak local perception and quadratic complexity. To promote the performance of attention mechanisms, we present a flexible multi-head linear attention (FMLA) architecture, which enhances locality awareness through layer-wise interactions with deformable convolutional blocks and online knowledge distillation. We develop a simple but effective mask mechanism that helps reduce noise influence in time series and reduces the redundancy of FMLA by probabilistically selecting and masking the positions of each given series. We use incremental ablation studies on 85 UCR2018 datasets to evaluate the performance of the main techniques developed. Experimental results demonstrate that FMLA outperformed 11 state-of-the-art TSC algorithms, obtaining a mean accuracy of 89.37%. FMLA achieved the best performance on 29 short-medium and 7 long time-series datasets regarding accuracy. FMLA has a complexity of ON, where N is the sample length. As N increases from 100 to 1000, the floating-point operations per second grow linearly from 0.13G to 1.34G.