LEBERT :Lite and Efficiently Optimized BERT PRetraining Approach

Abstract

The extensive generalization of these models can lead to overfitting, causing the model to perform poorly on unseen data and thereby not realizing its full potential. To address this challenge systematically, we propose a novel approach for lightweight and efficient fine-tuning of BERT (Bidirectional Encoder Representations from Transformers) that aims to achieve improved generalization and harness the maximum capabilities of BERT. Our proposed approach incorporates various regularization techniques designed to adaptively manage the model's complexity. We plan to conduct experiments using this approach across various NLP tasks, including GLUE (Wang et al., 2019), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016).