Multi-Head Attention in Residual Networks to Improve Coral Reef Structure Classification

Abstract

Residual Networks (ResNet) mark a crucial advancement in convolutional neural network architecture, effectively tackling challenges like vanishing gradients for improved pattern detection in various image classification tasks. This study introduces a novel adaptation of the ResNet50 architecture that integrates a multi-head attention mechanism (MHA), coined MHA-ResNet50, for discerning coral reef structures within images. Strategic modifications are applied to the input of each stage, leading to the development of an MHA block, which is augmented by separable convolution. The deliberate inclusion of the MHA block at various stages in identity-block Resnet50, in adherence to multiscale gate principles, precedes its traversal through fully connected layers. Furthermore, we implemented the Stratified K-fold concept to ensure that each fold has a comparable proportion of each class. We successfully assessed the efficacy of the MHA-Resnet50 model in several MHA-block placement scenarios and saw improvements in the accuracy of coral reef structure predictions. The most optimal results were achieved by incorporating four attention blocks (MHA-ResNet50-4), yielding an accuracy rate of 85.23% in recognition of coral structure images, comprising a mere 409 images. This model showcases adaptability to small datasets while delivering commendable performance. The ResNet50 architecture undergoes enhancement in our proposed model by integrating multi-head attention, separable convolution, and multiscale gate principles. The MHA-ResNet50 model substantially advances accurately predicting coral reef structures, demonstrating adaptability to limited datasets. Future lines of this research involve digging deeper into the model design and using more significant amounts and classes of data to strengthen a more comprehensive range of generalizations.