Abstract
Pre-trained language models such as BERT have proven essential in natural language processing(NLP). However, their huge number of parameters and training cost make them very limited in practical deployment. To overcome BERT’s lack of computing resources, we propose a BERT compression method by applying decoupled knowledge distillation and representation learning, compressing the large model(teacher) into a lightweight network(student). Decoupled knowledge distillation divides the classical distillation loss into target related knowledge distillation(TRKD) and non-target related knowledge distillation(NRKD). Representation learning pools the Transformer output of each two layers, and the student network learns the intermediate features of the teacher network. It has better results on tasks of Sentiment Classification and Paraphrase Similarity Matching, retaining 98.9% performance of the large model.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
