Ralali.com is a B2B e-commerce platform that offers various brands across categories ranging from automotive to building materials. The Play Store is a tool for downloading applications used by many people. This research aims to compare and find the best model among Naïve Bayes (NB), Support Vector Machine (SVM), and k-Nearest Neighbor (k-NN) in classifying the sentiment reviews of Ralali.com's application on the Play Store, and analyze the negative labels to provide recommendations for Ralali.com developers. Based on the research results, the NB Algorithm stands out as the best choice compared to SVM and k-NN in addressing class imbalance. The use of SMOTE generally improves the model performance on minority classes for Precision, Recall, and F-Measure, although there are still challenges related to the lower Accuracy compared to the use of non-SMOTE.