Near-infrared spectral interval screening based on hierarchical variables clustering and group SCAD in multivariate calibration

Abstract

Spectral interval screening is a critical step in multivariate calibration, which can improve the model predictive performance and data interpretation. In this study, a novel method for interval selection is proposed based on a hierarchical variables clustering and group smoothly clipped absolute deviation(group SCAD) in combination with partial least squares(VCG-PLS). The proposed method makes use of hierarchical variables clustering to yield a variables partitioning into groups at each level, and these groups of variables from different clustering levels are then used as input for group SCAD. The method is designed to select informative wavelength intervals for near-infrared(NIR) spectroscopic data analysis. The proposed method mainly consists of three steps. Firstly, an effective hierarchical clustering is employed to cluster wavelengths(variables), which generates a partition of variables into groups at each hierarchy level and obtains all possible wavelength intervals. Then, the series of group variables obtained from various hierarchy levels are given as input to group-SCAD, and group-SCAD can generate potential group variables corresponding to each regularization parameter value. Finally, a collection of PLS models is constructed recursively by employing all wavelength intervals except one, until the optimal wavelength intervals are obtained. The optimal intervals correspond to the lowest root mean square error of prediction. The VCG-PLS integrates the advantages of hierarchical variable clustering and group SCAD, which is an efficient technique to enhance the performance of PLS in interval selection. The performance of VCG-PLS was tested on three real NIR datasets. The results demonstrate that VCG-PLS can improve prediction performance with fewer variables and may be a good wavelength interval selection strategy.