Incorporating Measurement Uncertainty Into Machine Learning-Based Grade Predictions

Abstract

Abstract As part of the regulations published in October of 2019, PHMSA requires operators that do not have reliable records to conduct material verification in accordance with §192.607. As part of the material verification process, §192.607(d)(2) compels the operator to “[c]onservatively account for measurement inaccuracy and uncertainty using reliable engineering tests and analyses” when utilizing nondestructive examination (NDE) methods. The Pacific Gas and Electric Company (PG&E) has completed extensive testing to develop approaches that utilize nondestructive measurements to estimate grade. As part of this work, a supervised classification machine learning (ML) model was developed to predict pipe grade using NDE chemical composition measurements as inputs. While using the ML-based model provides substantial improvement over yield strength (YS) in predicting pipe grade, measurement uncertainty from NDE tools must be considered per §192.607(d)(2). Moreover, some amount of uncertainty is present in any measurement regardless of precision, and this measurement uncertainty may ultimately affect the ML model’s pipe grade classification. This paper presents a methodology for incorporating this variability into the authors’ ML classification model using a Monte Carlo-based simulation approach. In addition, this study will discuss the various metrics that were developed for interpreting the most probable pipe grade from the large number of simulation results, including the average probability, range of probability, and the number of simulations where each grade was identified as having the highest probability. Since any ML model can misclassify a sample and there are such slight differences between adjacent grades, it is necessary to have a method of systematically validating the results based on prior knowledge. Several case studies using field data will be presented to illustrate this approach, including validation cases where the pipe grade is known.