Assisting pre-delivery firmware quality assessments using ensemble learning

Abstract

ABSTRACT This study uses retrospective data for firmware tests as the input data sets to train four machine learning models with embedded standalone classifiers. None of these models provide accurate predictions during validation, so model optimization trials adjust the training-validation data portfolio and hyper parameters for each model. Consequently, only the random forest classifier with the best parametric settings just achieves the 90% prediction accuracy required by the standard. Ensemble learning (EL) is then applied using several combinations over the standalone models, and the EL model using logistic regression as the meta classifier increases the accuracy by 6% (i.e. to 96%), which is sufficient for establishing a predictive system. Using the ‘X-minute’ method, it is further identified that the execution period (also the data sampling period) for the sequential read test workload can be reduced from 30 (in current practice) to 20 minutes and that the predictions are sufficiently accurate for system implementation using the EL model. Applying the similarity confirmation method for each pair of ‘score vectors’ (each of which contains a model’s prediction accuracies), several observations distinguishing the performance and the predictive behavioral patterns of the benchmarked models are further confirmed. The knowledge from this advanced research has implications which may benefit future practice in industry.