A Novel Lightweight Language Model Architecture with Flexible Parameters

Abstract

In this paper, we introduce SwiFTeDLM, a groundbreaking Language Model architecture that leverages the power of SwiGLU for enhanced decoding capabilities. SwiFTeDLM stands for SwiGLU Enabled Fine-Tuned Decoder based Language Model, representing a fusion of state-of-the-art techniques in natural language processing. Our model achieves superior performance through the integration of SwiGLU, a recently developed activation function, enabling more effective information flow within the decoding mechanism. We conduct extensive experiments to demonstrate the effectiveness of SwiFTeDLM in various language tasks, showcasing its ability to challenge existing models. Additionally, we explore the fine-tuning aspect of the architecture, highlighting its adaptability to specific domains. SwiFTeDLM not only advances the field of language modeling but also opens avenues for further exploration and improvement in natural language understanding and generation. Also we have introduced a new pre-training method and further fine-tuned version of the model.