군 등변 컨벌루션을 사용하는 효율적인 군 등변 트랜스포머

Abstract

Self-attention is a permutation-equivariant operation by definition, and incorporating group-invariant positional encoding allows it to achieve equivariance for larger symmetry groups. We propose efficient group-equivariant transformers that eliminate the cubic complexity caused by group-invariant positional encoding. Instead of using explicit positional encoding, we replace linear projections with group-equivariant convolutional projections, which serve as both a group mixer and an implicit positional encoding. This allows for reducing the group of self-attention to the translation group while maintaining group equivariance, resulting in less computation and memory. A group-equivariant convolutional stem improves performance much more. The proposed method outperforms the existing group-equivariant transformer and CNNs on the rotated MNIST dataset, a standard dataset for evaluating group-equivariant networks.