Improving k-nearest neighbor performance using permutation feature importance to predict student success in study

Abstract

The timely graduation of students is a critical indicator of academic quality assessment. Therefore, universities should use effective predictive systems to identify earlier potential lateness of graduation. This study aimed to improve the K-nearest neighbor (K-NN) algorithm’s ability to predict student on-time graduation. It evaluated K-NN algorithm performance with and without the permutation feature importance (PFI) technique, using a dataset of 460 student graduation records from 2014 to 2017. The training data was oversampled, adjusting the ratio of minority class samples from 13% to 100% of the majority class samples. The result shows that integrating PFI into the K-NN model improved K-NN performance by 10 iterations of the PFI process, N-shuffle varying from 10 to 100 for each iteration, and a minority class sample ratio of 25%. The accuracy score improved from 90.22% to 92.39%, precision from 50.00% to 62.50%, F1-score from 52.63% to 58.82%, while recall remained consistent at 55.56%. The PFI analysis showed that achievement index for the 1st semester or IPS 1 had the least impact on the model. The study suggested using a comprehensive approach to determine the n-shuffle of PFI based on the number of test data for a more accurate feature contribution pattern.