Boosting the generalization ability of Vis-NIR-spectroscopy-based regression models through dimension reduction and transfer learning

Abstract

Calibration regression models based on visible and near-infrared (Vis/NIR) spectroscopy are now widely used in the rapid non-destructive prediction of agricultural products' quality parameters. However, the distributions of products' quality parameters and spectral responses are different in various batches of products, so a calibration regression model built on one dataset might be ineffective when tested in another. How to improve the generalization ability of models is a crucial problem and is formulated as ‘Calibration Transfer’. Calibration transfer was firstly proposed to eliminate the difference of spectral responses between spectrometers, but now it has been applied in the transfer between different domains of spectral samples or different components. In this paper, we proposed two robust models with great generalization ability in the calibration transfer task, respectively using dimension reduction and transfer learning, namely SPRS (Standard normal variate, Partial least squares dimension reduction, Ridge regression, Slope/bias) and SNV-based Aug-TrAdaBoost.R2. We tested the two models in spectral datasets of tea leaves to predict the moisture content of samples, it was found that SPRS and SNV-based Aug-TrAdaBoost.R2 can reach great performance over both source and target domains across different batches, different varieties, and different classes of tea leaf samples. SPRS and SNV-based Aug-TrAdaBoost.R2 achieved R2 values of 0.9314 and 0.9895 in cross-tea-class prediction whereas traditional calibration method PLSR + S/B only achieved 0.4874. SPRS had low computation complexity and was more robust while SNV-based Aug-TrAdaBoost.R2 had higher accuracy in target domain prediction but was computation-consuming. The two proposed models showed the potentials of online automatic quality parameters prediction and high-accuracy prediction across domains of various samples.