Evaluation of Stratified K-Fold Cross Validation for Predicting Bug Severity in Game Review Classification

Abstract

Steam review data provided a lot of information for the game development team, either positive or negative review. It contained an essential point that negative and positive reviews provide crucial information and 7% of positive reviews contained bug reports. These bug reports occurred after the game was released, and many reports of common incidents still exist. If players found an issue in the game, they could report it directly through the review feature provided by the online game platform. Nevertheless, the development team took a long time manually analyzing and categorizing the game review. This study proposed a new approach to automatically categorizing game reviews on Steam based on the bug severity level. Hence, to solve this problem, we suggested introducing a solution based on the research background indicated above. For this experiment, we analyzed reviews on two popular game titles namely, FIFA 23 and Apex Legends. We implemented three different classifiers namely, KNN, Decision Tree, and Naïve Bayes, which would be used to train a dataset to classify the bug severity level. Due to an imbalanced dataset, we performed cross-validation to reduce bias in the dataset. Performance in this model would be evaluated using accuracy rate, precision, recall, and F1 score. As a result, the experiment showed that game reviews of different game titles achieved different accuracy scores. The game review classification for FIFA 23 performed better than the game review classification for Apex Legends. The mean accuracy score of FIFA 23 was 72% with Decision Tree and Apex Legend was 64% with KNN.