Chapter 2 - Data Pre-processing

Abstract

Preprocessing techniques are designed to improve the linear relationship between the spectral signals and analyte concentrations. These techniques can be divided into two major groups that include those that directly use available reference values for the preprocessing operation and those that do not. The latter group is thus a reference-independent preprocessing group, and as such provides more general tools suitable for studies such as exploratory studies, where often no reference value is available. The reference-independent techniques can further be divided into two subgroups: scatter correction methods and derivation methods. Scatter correction methods include multiplicative signal correction (MSC), also known as multiplicative scatter correction (including extended MSC, inverse MSC, inverse extended MSC, and de-trending), standard normal variate (SNV) scaling, and normalization, and baseline correction. Derivation includes finite difference, Savitzky–Golay, and Norris–Williams, techniques. The basic concept of the MSC is to remove nonlinearities in the data caused by scatter from particulates in the samples. The MSC operation is divided into two steps: estimation of the correction coefficients, and correction of the spectra. There are two typical types of normalization used on spectral data that include total absolute sum and square root of the total squared sum (Euclidean norm).