Neutral-to-ground voltage (NTGV) abnormalities in secondary distribution systems (SDS) pose significant power quality (PQ) challenges, including safety hazards, power losses, and equipment damage. Despite their importance, these abnormalities remain relatively understudied. Accurate classification of NTGV events is crucial for effective mitigation strategies. Existing research primarily relies on machine learning (ML) models trained on manually extracted features from simulated or real-world signals. This paper introduces a novel end-to-end deep learning approach that leverages Gate Recurrent Units (GRU) to bypass manual feature extraction, directly utilizing real-world signals from three NTGV event categories: ground fault, lightning strike, and normal conditions. This is first time that GRU has been used for NTGV classification using raw data. The model's generalizability is assessed through 5-fold cross-validation. A comparative analysis with baseline models and traditional ML techniques demonstrates the proposed model's superior performance and computational efficiency due to its ability to directly process raw data.