Integrating self-attention mechanisms in deep learning: A novel dual-head ensemble transformer with its application to bearing fault diagnosis

Abstract

In this paper, we propose a novel dual-head ensemble Transformer (DHET) algorithm for the classification of signals with time–frequency features such as bearing vibration signals. The DHET model employs a dual-input time–frequency architecture, integrating a 1D Transformer model and a 2D Vision Transformer model to capture the spatial and time–frequency features. By utilizing data from both the time and time–frequency domains, the proposed algorithm broadens its feature extraction capabilities and enhances the model’s capacity for generalization. In our DHET structure, the original Transformer model leverages self-attention mechanisms to consider relationships among signal input segmentations, which makes it effective at capturing long-range dependencies in signal data, while the Vision Transformer model takes 2D images as input and creates the image patches for embedding and each patch is linearly embedded into a flat vector and treated as a ‘token,’ then the ‘tokens’ are processed by the Transformer layers to learn global contextual representations, enabling the model to perform signal classification task. This integration notably enhances the performance and capability of the model. Our DHET is especially effective for rolling bearing fault diagnosis. The simulation results show that the proposed DHET has higher classification accuracy for bearing fault diagnosis and outperforms CNN-based methods.