Identifying Reply-to Relation in Textual Group Chat using Unlabeled Dialogue Scripts and Next Sentence Prediction

Abstract

With instant message (IM) software becoming an important part of work and study, more and more research has begun to aim at supporting people's communication by analyzing their chat messages, such as topic provision and relationship sustainment. As the essential step for achieving these research works, the first task is to identify the relations between those large amounts of chat messages, especially in the situation of group chats. In this paper, we propose a method to identify the “reply-to” relations in IM's group chat from unlabeled textual data by using the next sentence prediction (NSP) approach. First, we proposed a method of automatically sampling two messages with and without the “reply-to” relation from unlabeled dialogue scripts to prepare the training data. Second, we built and trained three settings of the NSP model through the training data to identify the “reply-to” relations between two text chat messages. These NSP models were based on the pre-trained Japanese BERT (bidirectional encoder representation from transformers) model. Last, we evaluated the trained models through actual text group chat data with manual labels. The evaluation data contains 444 textual chat messages from three chat groups, each group has three chat members. Evaluation results showed that the models reached a max accuracy up to 69.6%, higher than past methods, and the top F1 score is 0.558.