Road damages, such as potholes and rutting, must be addressed in their early stages to prevent accidents and minimize maintenance costs. The presence of these damages not only causes discomfort to passengers but also accelerates vehicle deterioration. Improper material mixtures and inadequate maintenance practices often contribute to road damage. In recent years, numerous machine learning (ML) algorithms have been developed with a predominant focus on binary classification, specifically targeting potholes. However, roads are susceptible to various damages beyond potholes, necessitating a comprehensive approach for effective solutions. This study introduces a novel algorithm utilizing a random forest (RF) classifier for multi-class classification of road damages. The proposed algorithm is validated through rigorous simulation and field studies. Unlike previous models, this approach embraces a comprehensive perspective by considering a wider range of damages, thereby facilitating a more nuanced and inclusive process for the classification of road damages. In the field study, data has been collected using a set of uni-axial accelerometers and a smartphone camera mounted on a test vehicle. The data has been processed using the sliding window technique, and features have been extracted from each window to train the RF classifier. In this process, an optimal window size has been determined and employed to enhance the effectiveness of feature extraction for training the RF classifier. The proposed algorithm demonstrates significant accuracy in both simulation and field studies, with notable performance in identifying and classifying road damages. The algorithm’s outcomes are utilized to estimate a detailed cost for repairing the identified damages. This paper showcases proficiency in addressing various damages, offering valuable insights for the implementation of cost-effective road maintenance strategies.