Pavement roughness monitoring has always been a concern in the field of road asset management. However, the utilization of laser profilometers to measure pavement roughness is costly and inconvenient. To address these challenges, this study developed a low-cost, lightweight, and rapid approach for pavement roughness estimation using smartphone sensor data. Firstly, the feasibility of the chosen smartphone in vibration acceleration acquisition was examined through micro-scale shaking table tests. Then, the Categorical Boosting (CatBoost) model with sequential model-based optimization (SMBO) approach was developed for smartphone-based pavement roughness estimation. Finally, the feature importance and feature interaction effects in the pavement roughness estimation task were interpreted using Shapley Additive exPlanations (SHAP) and Partial Dependence Plots (PDP), respectively. The results show that the chosen smartphone has great potential in vibration data acquisition, and the recorded data are highly consistent with the results obtained by the professional accelerometer. The CatBoost model outperforms the other reference models in terms of pavement roughness estimation accuracy, with coefficient of determination (R2) of 0.807, root mean square error (RMSE) of 0.553, and mean absolute error (MAE) of 0.433. Furthermore, the feature interpretation results indicate that the smartphone-based pavement roughness estimation approach relies on the coupling effects of vibration and velocity features. This study provides a novel insight into the lightweight pavement roughness estimation approach, which has potential implications for assisting pavement maintenance decision-making in road asset management.