Use all tokens method to improve semantic relationship learning

Abstract

Recently, research on inference methods has been actively conducted to use language models more effectively for studying natural language understanding. Inference in language models that use bidirectional encoder representations from transformers (BERT) is performed using classification tokens that convey information from the input sentences. The use of single-token inference method for inference does not involve the hidden state vector that contains relevant connection information between the words, which in turn limits the ability to infer semantic relationships. This study proposes a use all tokens (UAT) method that combines unused tokens to improve inference methods through a single token. The UAT method effectively combines hidden state vectors and ensembles the global information of sentences with the local information between words. When the Stanford natural language inference (SNLI) corpus was solved using DeBERTaV3large, compared to the existing single token inference method, the UAT method improved the precision of the neutral relationship by 4.3% (87.7% vs. 92.0%) and the recall of the entailment and contradiction relationship by an average of 2% (93.5% vs. 95.5%). The UAT method proposed in this study can be readily implemented in BERT-based language models, and it enhances the accuracy and F1-score, thereby improving the learning of semantic relationships between sentences.