Transforming Text Generation in NLP: Deep Learning with GPT Models and 2023 Twitter Corpus Using Transformer Architecture

Abstract

This paper presents the design, implementation, and evaluation of a Transformer-based Generative Pre-trained Transformer (GPT) model tailored for character-level text generation. Leveraging the robust architecture of the Transformer, the model has been trained on a corpus sourced from social media text data, with the aim of exploring the intricacies of language patterns within a condensed and informal text setting. Key aspects of the model include a multi-head self-attention mechanism with a custom head configuration, positional embeddings, and layer normalization to promote stability in learning. It operates with a defined set of hyperparameters: a batch size of 32, a block size of 128, 200 iterations, a learning rate of 3e-4, and employs 4 attention heads across 4 layers with an embedding dimension of 384. The model has been optimized using the AdamW optimizer and includes regularization through dropout to prevent overfitting.Through a series of training iterations, the model demonstrates a converging behavior in loss metrics, indicating effective learning, and showcases the capacity to generate coherent text sequences post-training. Training and validation losses have been reported, revealing the nuances in model performance and generalization capabilities. The generated text samples postulate the model's potential in capturing the contextual flow of the dataset. This study further plots the loss curves, visually representing the training dynamics and convergence patterns. The final model, encapsulated within a PyTorch framework, presents a step forward in the realm of neural text generation, contributing to the ongoing advancements in language modeling and its applications in understanding and generating human-like text.