BERT-Caps: A Transformer-Based Capsule Network for Tweet Act Classification

Abstract

Identification of speech acts provides essential cues in understanding the pragmatics of a user utterance. It typically helps in comprehending the communicative intention of a speaker. This holds true for conversations or discussions on any fora, including social media platforms, such as Twitter. This article presents a novel tweet act classifier (speech act for Twitter) for assessing the content and intent of tweets, thereby exploring the valuable communication among the tweeters. With the recent success of Bidirectional Encoder Representations from Transformers (BERT), a newly introduced language representation model that provides pretrained deep bidirectional representations of vast unlabeled data, we introduce BERT-Caps that is built on top of BERT. The proposed model tends to learn traits and attributes by leveraging from the joint optimization of features from the BERT and capsule layer to develop a robust classifier for the task. Some Twitter-specific symbols are also included in the model to observe its influence and importance. The proposed model attained a benchmark accuracy of 77.52% and outperformed several strong baselines and state-of-the-art approaches.