Data engineering for predictive machine learning of stormwater infrastructure conditions

Abstract

Culverts are stormwater structures that require ongoing condition assessment and proactive maintenance for optimal performance to reduce flooding, especially in urban areas. Managing these structures can be complex and costly regarding time, money, and personnel resources. Machine learning (ML) tools are a powerful means of predicting culvert conditions via learning from their existing data records. However, the data available on such subjects are usually low in quantity and of various qualities; thus, they need significant data processing for ML. Existing stormwater infrastructure condition prediction studies rarely detail their data processing methods. Hence, this work uses a comprehensive case study to illustrate essential data preparation techniques to address common data issues and enhance the performance of commonly used ML algorithms. The study shows methods for exploratory data analysis to understand the dataset, data wrangling methods for preprocessing data of various quality, and data engineering procedures for addressing insufficient data issues. After the data processing procedures are applied, F-1 scores are used to evaluate the ML models' performance. The random forest classifier model, one of the four ML models, performed best after applying the data wrangling and data engineering methods. Overall, this work provides transparency of methods and applications to encourage ML use in the water resources engineering field.