A novel severity calibration algorithm for defect detection by constructing maps

Abstract

In industrial scenarios, knowledge of the severity of defects is needed to distinguish different levels of defects and adapt to the possible modification of defect identification criteria. Well-calibrated detection models are crucial, especially for these decision-making applications. Previous works have measured and calibrated biased confidence estimates of classification and object detection models, while the field of severity calibration has not yet been addressed. Through experiments, we find that the severity also has a specific relationship with the probabilistic prediction of models. Therefore, we present a framework to measure and calibrate severity calibration for defect detection applications by constructing maps between probabilistic prediction, precision, and severity. We find that unlike confidence calibration, the reliability diagram of a well-calibrated model for severity calibration is funnel-shaped instead of linear-shaped. To approximate the funnel-shaped mapping function, we introduce a Gaussian distribution -based function with parameters that are fitted by marked data. Additionally, extending algorithms are proposed to adjust classical confidence calibration methods (e.g., histogram binning, isotonic regression, vector scaling, and temperature scaling) for severity calibration. By mapping the probabilistic prediction to precision, severity calibration is transformed to confidence calibration. The proposed severity calibration methods are evaluated by the expected severity calibration error (ESCE) in comparative experiments, which show that the extended isotonic regression achieves the best performance. In Zhongce’s 21 intelligent detection lines, we applied severity calibration and received good feedback from inspectors.