The influence of training data variability on a supervised machine learning classifier for Structure from Motion (SfM) point clouds of rock slopes

Abstract

Supervised Machine Learning (ML) can be used to automatically interpret remote sensing data in engineering geology, with applications for rockfall and landslide characterization. However, supervised algorithms typically require very large training databases from which to learn predictive relationships, and there is little guidance on how to construct such a database in an earth science context with high temporal and spatial heterogeneity. This study builds a supervised classifier to perform basic rock slope characterization on Structure from Motion (SfM) point clouds collected under a variety of conditions. Eight datasets were collected in Colorado and Utah, USA, with multiple different sensor platforms (terrestrial and aerial), rock types, seasons, and lighting conditions, and each dataset was manually labeled to identify regions of vegetation, rock, soil, talus, and snow. A total of 2560 Random Forest models were built with different combinations of training datasets and combinations of geometric and color features. Most models were able to identify vegetation and rock with high accuracy (median F scores of 86% and 68% respectively), but performance for soil, talus, and snow was overall much poorer, and the median overall accuracy of generalized classifiers was 60%. Many characteristics of the training data were found to have significant effects on generalization accuracy, indicating that training datasets must be curated to be applicable to specific data collection parameters, seasons, lighting conditions, and geological settings. We conclude that high accuracy generalized results can be obtained, but the ML model must be carefully constructed, and its limitations acknowledged.