GRA-Net: Group response attention for deep learning

Abstract

Activation function, one of the most critical components in deep learning, enables the artificial neural networks to learn complex patterns through nonlinearity. Currently, element-wise activation functions such as ReLU are widely utilized because of their simplicity and efficiency. However, these methods do not consider the interactions among neurons, which may produce redundant features and affect performance. Inspired by neuroscience, we propose Group Response Attention (GRA). It establishes connections among neurons, which means that the state of each member is determined by the entire group. Accordingly, GRA makes the features more diverse and reduces redundancy. In addition, we propose a pure attention network without any conventional activation function, termed as GRA-Net. Experiments conducted on typical computer vision tasks show that GRA-Net achieves state-of-the-art results. For the ImageNet-1K classification task, GRA-Net-T achieves 82.9% Top-1 accuracy. With comparable parameters and FLOPs, its performance improves by 1.6% and 0.8% over Swin-T and ConvNeXt-T. On the COCO detection task, as the backbone of Mask R-CNN, GRA-Net-T outperforms Swin-T and ConvNeXt-T by 2.1% and 2.0% on box AP. Meanwhile, the mIoU of GRA-Net-T on the ADE20K semantic segmentation task is higher than Swin-T and ConvNeXt-T by 1.9% and 1.0%.