Abstract
Recently, research on inference methods has been actively conducted to use language models more effectively for studying natural language understanding. Inference in language models that use bidirectional encoder representations from transformers (BERT) is performed using classification tokens that convey information from the input sentences. The use of single-token inference method for inference does not involve the hidden state vector that contains relevant connection information between the words, which in turn limits the ability to infer semantic relationships. This study proposes a use all tokens (UAT) method that combines unused tokens to improve inference methods through a single token. The UAT method effectively combines hidden state vectors and ensembles the global information of sentences with the local information between words. When the Stanford natural language inference (SNLI) corpus was solved using DeBERTaV3large, compared to the existing single token inference method, the UAT method improved the precision of the neutral relationship by 4.3% (87.7% vs. 92.0%) and the recall of the entailment and contradiction relationship by an average of 2% (93.5% vs. 95.5%). The UAT method proposed in this study can be readily implemented in BERT-based language models, and it enhances the accuracy and F1-score, thereby improving the learning of semantic relationships between sentences.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.