Evaluation of classification models in limited data scenarios with application to additive manufacturing

Abstract

This paper presents a novel framework that enables the generation of unbiased estimates for test loss using fewer labeled samples, effectively evaluating the predictive performance of classification models in data-limited applications. The framework’s key innovation lies in developing an adaptive sampling distribution that iteratively identifies influential testing samples based on interactions between learner and evaluator agents. Notably, the adaptive distribution dynamically adjusts the evaluator agent’s supervisory role by prioritizing inputs with discrepancies between the agents and considering the evaluator’s uncertainty. Comprehensive experimental analyses on synthetic data and two sparse data sets from material extrusion additive manufacturing problems validate the framework’s superiority over uniform and fixed sampling distributions. First, the proposed framework provides unbiased estimates of the test loss across various data sets, sampling ratios, and evaluator models. Second, the introduced adaptive sampling distribution significantly reduces the standard deviation of the test loss estimator compared to uniform sampling, achieving a 50% reduction for a 10% sampling ratio in the filament selection benchmark. Third, the framework demonstrates its efficacy in model selection to determine the optimal number of hidden units with a reduced number of test samples. Overall, this work offers a promising framework for evaluating classification models in applications where acquiring labeled data is time-consuming and resource-intensive, including materials science and engineering.