Abstract
The transformers architecture and transfer learning have radically modified the Natural Language Processing (NLP) landscape, enabling new applications in fields where open source labelled datasets are scarce. Space systems engineering is a field with limited access to large labelled corpora and a need for enhanced knowledge reuse of accumulated design data. Transformers models such as the Bidirectional Encoder Representations from Transformers (BERT) and the Robustly Optimised BERT Pretraining Approach (RoBERTa) are however trained on general corpora. To answer the need for domain-specific contextualised word embedding in the space field, we propose SpaceTransformers, a novel family of three models, SpaceBERT, SpaceRoBERTa and SpaceSciBERT, respectively further pre-trained from BERT, RoBERTa and SciBERT on our domain-specific corpus. We collect and label a new dataset of space systems concepts based on space standards. We fine-tune and compare our domain-specific models to their general counterparts on a domain-specific Concept Recognition (CR) task. Our study rightly demonstrates that the models further pre-trained on a space corpus outperform their respective baseline models in the Concept Recognition task, with SpaceRoBERTa achieving significant higher ranking overall.
Highlights
In the past three years, the transformers architecture [1] and transfer learning [2] have profoundly impacted the Natural Language Processing (NLP) landscape
Transfer learning consists of two stages: (i) a pre-training phase in which contextualised word embeddings are learned through selfsupervised training tasks on a large unlabelled corpus (for instance, Masked Language Model (MLM) and Sentence Prediction (NSP) [2]), and (ii) a second phase in which the pre-trained model is fine-tuned for a specific task [3]
The performance of the downstream NLP tasks are greatly improved with the knowledge transferred from the pre-trained models
Summary
In the past three years, the transformers architecture [1] and transfer learning [2] have profoundly impacted the Natural Language Processing (NLP) landscape. Transfer learning consists of two stages: (i) a pre-training phase in which contextualised word embeddings are learned through selfsupervised training tasks on a large unlabelled corpus (for instance, Masked Language Model (MLM) and Sentence Prediction (NSP) [2]), and (ii) a second phase in which the pre-trained model is fine-tuned for a specific task [3]. The performance of the downstream NLP tasks are greatly improved with the knowledge transferred from the pre-trained models. Numerous studies presented the theoretical background and empirical proof of the positive impact of the pre-training and fine-tuning setting for downstream tasks [4, 5]. Where ti is the ith word of the sequence These tokens have a fixed initial embedding of dimension n, noted as xi. The pre-trained model is fine-tuned for a specific task.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
