Standard Normal Variate And Detrending Research Articles

Combination of Hyperspectral and Machine Learning to Invert Soil Electrical Conductivity

An accurate estimation of soil electrical conductivity (EC) using hyperspectral techniques is of great significance for understanding the spatial distribution of solutes and soil salinization. Although spectral transformation has been widely used in data pre-processing, the performance of different pre-processing techniques (or combination methods) on different models of the same data set is still ambiguous. Moreover, extremely randomized trees (ERT) and light gradient boosting machine (LightGBM) models are new learning algorithms with good generalization performance (soil moisture and above-ground biomass), but are less studied in estimating soil salinity in the visible and near-infrared spectra. In this study, 130 soil EC data, soil measured hyperspectral data, topographic factors, conventional salinity indices such as Salinity Index 1, and two-band (2D) salinity indices such as ratio indices, were introduced. The five spectral pre-processing methods of standard normal variate (SNV), standard normal variate and detrend (SNV-DT), inverse (1/OR) (OR is original spectrum), inverse-log (Log(1/OR) and fractional order derivative (FOD) (range 0–2, with intervals of 0.25) were performed. A gradient boosting machine (GBM) was used to select sensitive spectral parameters. Models (extreme gradient boosting (XGBoost), LightGBM, random forest (RF), ERT, classification and regression tree (CART), and ridge regression (RR)) were used for inversion soil EC and model validation. The results reveal that the two-dimensional correlation coefficient highlighted EC more effectively than the one-dimensional. Under SNV and the second order derivative, the two-dimensional correlation coefficient increased by 0.286 and 0.258 compared to the one-dimension, respectively. The 13 characteristic factors of slope, NDI, SI-T, RI, profile curvature, DOA, plane curvature, SI (conventional), elevation, Int2, aspect, S1 and TWI provided 90% of the cumulative importance for EC using GBM. Among the six machine models, the ERT model performed the best for simulation (R2 = 0.98) and validation (R2 = 0.96). The ERT model showed the best performance among the EC estimation models from the reference data. The kriging map based on the ERT simulation showed a close relationship with the measured data. Our study selected the effective pre-processing methods (SNV and the 2 order derivative) using one- and two-dimensional correlation, 13 important factors and the ERT model for EC hyperspectral inversion. This provides a theoretical support for the quantitative monitoring of soil salinization on a larger scale using remote sensing techniques.

Combination of efficient signal pre-processing and optimal band combination algorithm to predict soil organic matter through visible and near-infrared spectra.

Visible and near-infrared (Vis-NIR) spectroscopy is a promising alternative to replace soil physicochemical Analysis to quickly and effectively determine the content of soil organic matter (SOM). However, choosing appropriate pre-processing methods and effective data mining techniques is the essential step in Vis-NIR to improve the quality of spectral data and the accuracy of the model prediction. In this study, nine spectral pre-processing methods and optimal band combination algorithms were introduced to process the spectra and select sensitive spectral parameters. The purpose of this study is to determine the effective pre-processing method and explore the prediction potential of the optimal band combination algorithm. Two hundred thirty-three soil samples were gathered from northwestern Xinjiang, China, and the soil properties and reflectance spectra were measured in the laboratory. The spectra were subjected to nine pre-processing methods, e.g., Savitzky-Golay (SG) smoothing, discrete wavelet transformation (DWT), First (FD) and second (SD) derivatives, multiplicative scatter correction (MSC), standard normal variate and detrend (SNV-DT), continuum removal (CR), correction by the maximum reflectance (CMR) and pseudo-absorbance values and detrend (Abs-DT). The results indicate, the SG proved to be the most effective pre-processing method for SOM in saline soil. The Abs-DT, FD, SD, SNV-DT, MSC, CR, DWT, and CMR led to degrading the prediction performance. Furthermore, the use of SG before further processing can improve the prediction effect, although it is not obvious. The optimal band combination algorithm can derive spectral parameters that have a good correlation with SOM content. Prediction accuracy (RPIQ was 3.058 and 3.045 in independent and cross-validation respectively) and model complexity (latent variables were both 4) from spectral parameter combination were both better than that from full-spectrum data. In summary, the combination of SG and the optimal band combination algorithm can improve the prediction accuracy of SOM in saline soil.

Detection of Three-component Adulteration in Camellia Oleifera Abel Seed Oil by Near Infrared Spectroscopy

In this paper, A combination of near-infrared microscopy and chemometrics were employed to identify the adulteration in Camellia oleifera Abel seed oil. The qualitative and quantitative detection methods for three-component adulteration in Camellia oleifera Abel seed oil was studied, and the models for the authenticity identification and content determination of Camellia oleifera Abel seed oil were established. Multivariate scattering correction (MSC), standard normal variate (SNV) and combination of standard normal variate and de-trend (SNV-DT) were separately used for the preprocessing of original spectra. Backward interval partial least squares (BiPLS) method was used to optimize feature variables, and partial least squares (PLS) method was employed to extract the spectrum from the characteristic wavelength. The parameters were optimized through comparative interactive verification (CV) and genetic algorithm (GA). With the establishment of qualitative and quantitative models in the full wavelength and characteristic wavelength ranges, the qualitative and quantitative identification paths were optimized. The results show that the identification and quantitative detection of Camellia oleifera Abel seed oil adulterated with three dopants could be realized by NIRS. The accuracy of as-established MSC-BiPLS-GA-SVC model was up to 100%. The correlation coefficient R of the MSC-BiPLS-GA-SVC model reached 99.7254%, and the MSE value was 0.0234.

Near-infrared-based Identification of Walnut Oil Authenticity

Near-infrared spectroscopy (NIR) was employed to determine the quality of walnut oil. Multivariate scattering correction (MSC) and combination of standard normal variate and de-trend (SNV-DT) were separately used for preprocessing. Successive projection algorithm (SPA) and combination of competitive adaptive reweighted sampling algorithm and partial least squares method (CARS-PLS) were separately used for extraction of characteristic wavelengths. Particle swarm optimization (PSO) and genetic algorithm (GA) methods were separately used for parameter optimization. Eventually, quantitative prediction models for impurities contents in walnut oil were established by support vector machine (SVM). After analysis of the prediction results derived from these models, an optimal candidate was obtained.

Rapid analyses of dry matter content and carotenoids in fresh cassava roots using a portable visible and near infrared spectrometer (Vis/NIRS).

Portable Vis/NIRS are flexible tools for fast and unbiased analyses of constituents with minimal sample preparation. This study developed calibration models for dry matter content (DMC) and carotenoids in fresh cassava roots using a portable Vis/NIRS system. We examined the effects of eight data pre-treatment combinations on calibration models and assessed calibrations on processed and intact root samples. We compared Vis/NIRS derived-DMC to other phenotyping methods. The results of the study showed that the combination of standard normal variate and de-trend (SNVD) with first derivative calculated on two data points and no smoothing (SNVD+1111) was adequate for a robust model. Calibration performance was higher with processed than the intact root samples for all the traits although intact root models for some traits especially total carotenoid content (TCC) (R2c = 96%, R2cv = 90%, RPD = 3.6 and SECV = 0.63) were sufficient for screening purposes. Using three key quality traits as templates, we developed models with processed fresh root samples. Robust calibrations were established for DMC (R2c = 99%, R2cv = 95%, RPD = 4.5 and SECV = 0.9), TCC (R2c = 99%, R2cv = 91%, RPD = 3.5 and SECV = 2.1) and all Trans β-carotene (ATBC) (R2c = 98%, R2cv = 91%, RPD = 3.5 and SECV = 1.6). Coefficient of determination on independent validation set (R2p) for these traits were also satisfactory for ATBC (91%), TCC (88%) and DMC (80%). Compared to other methods, Vis/NIRS-derived DMC from both intact and processed roots had very high correlation (>0.95) with the ideal oven-drying than from specific gravity method (0.49). There was equally a high correlation (0.94) between the intact and processed Vis/NIRS DMC. Therefore, the portable Vis/NIRS could be employed for the rapid analyses of DMC and quantification of carotenoids in cassava for nutritional and breeding purposes.

Estimating the soil clay content and organic matter by means of different calibration methods of vis-NIR diffuse reflectance spectroscopy

The selection of calibration method is one of the main factors influencing measurement accuracy of soil properties estimation in visible and near infrared reflectance spectroscopy. In this study, the performance of three regression techniques, namely, partial least-squares regression (PLSR), support vector regression (SVR), and multivariate adaptive regression splines (MARS) were compared to identify the best method to assess organic matter (OM) and clay content in the salt-affected soils. One hundred and two soil samples collected from Northern Sinai, Egypt, were used as the data set for the calibration and validation procedures. The dry samples were scanned using a FieldSpec Pro FR Portable Spectroradiometer (Analytical Spectral Devices, ASD) with a measurement range of 350–2500nm. The spectra were subjected to seven pre-processed techniques, e.g., Savitzky–Golay (SG) smoothing, first derivative with SG smoothing (FD-SG), second derivative with SG smoothing (SD-SG), continuum removed reflectance (CR), standard normal variate and detrending (SNV-DT), multiplicative scatter correction (MSC) and extended MSC. The results of cross-validation showed that in most cases MARS models performed better than PLSR and SVR models. The best predictions were obtained using MARS calibration methods with CR prep-processing, yielding R2, root mean squared error (RMSE), and ratio of performance to deviation (RPD) values of 0.85, 0.19%, and 2.63, respectively, for OM; and 0.90, 5.32%, and 3.15, respectively, for clay content.

Two-dimensional correlation spectroscopy (2D-COS) variable selection for near-infrared microscopy discrimination of meat and bone meal in compound feed.

This article presents a novel method for combining auto-peak and cross-peak information for sensitive variable selection in synchronous two-dimensional correlation spectroscopy (2D-COS). This variable selection method is then applied to the case of near-infrared (NIR) microscopy discrimination of meat and bone meal (MBM). This is of important practical value because MBM is currently banned in ruminate animal compound feed. For the 2D-COS analysis, a set of NIR spectroscopy data of compound feed samples (adulterated with varying concentrations of MBM) was pretreated using standard normal variate and detrending (SNVD) and then mapped to the 2D-COS synchronous matrix. For the auto-peak analysis, 12 main sensitive variables were identified at 6852, 6388, 6320, 5788, 5600, 5244, 4900, 4768, 4572, 4336, 4256, and 4192 cm(-1). All these variables were assigned their specific spectral structure and chemical component. For the cross-peak analysis, these variables were divided into two groups, each group containing the six sensitive variables. This grouping resulted in a correlation between the spectral variables that was in accordance with the chemical-component content of the MBM and compound feed. These sensitive variables were then used to build a NIR microscopy discrimination model, which yielded a 97% correct classification. Moreover, this method detected the presence of MBM when its concentration was less than 1% in an adulterated compound feed sample. The concentration-dependent 2D-COS-based variable selection method developed in this study has the unique advantages of (1) introducing an interpretive aspect into variable selection, (2) substantially reducing the complexity of the computations, (3) enabling the transferability of the results to discriminant analysis, and (4) enabling the efficient compression of spectral data.

The Effect of Regions of Interest and Spectral Pre-Processing on the Detection of Non-0157 Shiga-Toxin Producing Escherichia Coli Serogroups on Agar Media by Hyperspectral Imaging

Foodborne infection caused by Shiga toxin-producing Escherichia coli (STEC) is a major worldwide health concern. The best known and highly virulent STEC serogroup is E. coli 0157:H7, which can be easily identified when cultured on sorbitol-MacConkey (SMAC) agar. Recently, six non-0157 STEC serogroups have been found to cause human illnesses. These non-0157 serogroups ferment sorbital and form pink colonies; therefore SMAC agar cannot be used to differentiate non-0157 serogroups from each other and other flora growing on the plate. This study investigated the potential of visible and near infrared hyperspectral imaging and chemometrics to spectrally differentiate six representative non-0517 STEC serogroups (026, 045, 0103, 0111, 0121 and 0145) grown as spots on Rainbow agar media. Mahalanobis distance classifiers were developed with spectra obtained from ground truth regions of interest (ROIs) of each serogroup colony. The ROIs were selected as a doughnut-like open-ellipse to only include the leading edge of growth and as a closed-ellipse covering the entire colony. For each ROI type, the Mahalanobis distance classifiers were developed with log (1/Reflectance), first derivative and standard normal variate and detrending (SNVD) pre-processing treatments. Serogroups 045 and 0121 were consistently classified over 98% accurate, regard less of the classification algorithm used. The lowest classification accuracies were from classifiers developed with only log (1/ R) ROI spectra. First derivative and SNVD spectra helped to increase the detection accuracies of the other serogroups. The classification accuracy for serogroups 026, 0111, 0103 and 0145 with the closed-ellipse and open-ellipse classification algorithms showed varying results from 8% to 87% and 57% to 100%, respectively. The lower accuracies with closed ellipse spectra were due to greater spectral variation in the centre pixels on a per-pixel basis. Practical implications of this study are the demonstrated potential of hyperspectral imaging for presumptive-positive screening of non-0157 serogroups on Rainbow agar and the extensibility of the developed sampling methods and classification models for future research to identify the target bacteria in the presence of background flora grown on spread plates.

Near Infrared Reflectance Calibration Optimisation to Predict Lignocellulosic Compounds in Sugarcane Samples with Coarse Particle Size

Frequent variations in spectral intensity due to particle size and/or of particle size distribution are observed in plant products processed in powder form and scanned with near infrared reflectance (NIR). In this study, two grinders, with differences in time consumption, practicality and providing homogenates with different particle size range and distribution, were tested to evaluate their effects on NIR spectra. Optimisation of NIR calibration was necessary before predicting lignocellulosic compounds in sugarcane ( Saccharum spp.) samples with coarse particle size to supply a pre-existing ecophysiological growth model. Sixty samples from three varieties, grown in four contrasting pedoclimatic areas and from five anatomical parts were scanned and then analysed by biochemical fractionation. Different calibration methods, resulting in a combination of multiple linear regressions (MLR) applied to three calibration sets (fine, coarse and mixed particle sizes) treated with six data pretreatments—first derivative (D), second derivative (D2), multiplicative scatter correction (MSC), standard normal variate and detrend (SNVD), standard normal variate and detrend successively followed by first derivative (SNVD-D) or second derivative (SNVD-D2)—were investigated. The best NIR model statistical values were obtained by calibration developed on a mixed calibration set treated by SNVD-D2. Results confirmed that NIR spectroscopy could be an accurate and efficient method to predict lignocellulosic compounds in different botanical parts of sugarcane samples when used as input to an ecophysiological growth model.

Application of near infrared spectroscopy for rapid detection of aflatoxin B1 in maize and barley as analytical quality assessment

The establishment of fast and non-destructive methods for the evaluation of quality and safety of raw grains is being demanded nowadays to avoid toxic substance presence. Aflatoxin B1 (AFB1) has been recognised by the International Agency of Research on Cancer as a group 1 carcinogen for animals and humans and the EU Official Journal has established action levels for AFB1 presence in all feed materials between 5 and 20 ppb. Near infrared spectroscopy (NIRS) is an excellent candidate for a rapid and low-cost method for the detection of aflatoxins in cereals. This study assesses the utility of NIRS for rapid detection of mycotoxigenic fungi as AFB1. A total of 152 samples were involved and analysed for aflatoxin content. The results of spectroscopic models developed have demonstrated that NIRS technology is an excellent alternative for fast AFB1 detection in cereals. The best predictive model to detect AFB1 in maize was obtained using standard normal variate and detrending (SNVD) as scatter correction ( r 2 = 0.80 and 0.82; SECV = 0.211 and 0.200 for grating and FT-NIRS instruments, respectively). In the case of barley, the best predictive model was developed using SNVD on the dispersive NIRS instrument ( r 2 = 0.85 and SECV = 0.176) and using spectral data as log 1/ R for FT-NIRS ( r 2 = 0.84 and SECV = 0.183).

Application of near-infrared microscopy (NIRM) for the detection of meat and bone meals in animal feeds: A tool for food and feed safety

This paper reports on the development and validation of a method for detecting meat and bone meal (MBM) in compound feeds by near-infrared reflectance microscopy (NIRM) as an alternative in food and feed safety. A FT-NIR (Fourier transformer-near-infrared reflectance) instrument attached to a microscope was used to build up a spectral library containing reference feed particles identified as plant or animal origin, from various sources. Spectra were collected directly from particles in the NIR spectrum region (1112–2500 nm). The spectral library sample set was used to develop various discriminant models to classify spectra as MBM or plant material. The best discriminant model was obtained using partial least squares (PLS) discriminant analysis and standard normal variate and detrending (SNVD) and first derivative for spectrum pretreatment; this model had a coefficient of determination of 0.95 and a standard error of cross-validation of 0.133. The model was externally validated. The results confirmed NIRM as a valuable technique for detection of banned MBM.

Measurement of chemical composition in wet whole maize silage by visible and near infrared reflectance spectroscopy

Visible (Vis) and near infrared reflectance (NIR) spectroscopy were used to predict dry matter (DM), acid and neutral detergent fibre (ADFom and aNDFom), ash, crude protein (CP) and pH in wet whole maize (WWM) silage samples. Samples were analysed by reference methods and spectra collected using a NIR spectrophotometer in reflectance (400–2500 nm). Predictive equations were developed using modified partial least squares (MPLS) with full cross validation using raw spectra or second derivative. Two scatter corrections were also used, namely standard normal variate and detrend (SNVD) and multiplicative scatter correction (MSC). Coefficient of determination in calibration R cal 2 and standard error of cross validation (SECV) were 0.91 (SECV: 6.5 g kg −1 DM) for CP, 0.85 (SECV: 27.4 g kg −1) for DM using SNVD, and 0.86 (SECV: 22.1 g kg −1 DM) for ADFom using raw spectra. Poor correlations between the reference methods and NIR were found for aNDFom, ash, OMD and pH. It was concluded that NIR spectroscopy might be a suitable method to predict DM, CP and ADFom on WWM silage samples with minimal sample preparation.

Non-destructive prediction of chemical composition in sunflower seeds by near infrared spectroscopy

Near infrared reflectance spectroscopy (NIRS) was explored as a technique to predict moisture (M), oil and crude protein (CP) content on intact sunflower seeds ( Helianthus annuus L.). Three hundred samples were scanned intact in a monochromator instrument NIRS 6500 (NIRSystems, Silver Spring, MD, USA). Calibration equations were developed using modified partial least square regression (MPLS) with internal cross validation. Samples were split in two sets, one set used as calibration ( n=250) where the remaining samples ( n=50) were used as validation set. Two mathematical treatments (first and second derivative), none (log 1/ R) and standard normal variate and detrend (SNVD) as scatter corrections were explored. The coefficient of determination in calibration ( R cal 2) and the standard error in cross validation (SECV) were 0.95 (SECV: 3.3) for M; 0.96 (SECV: 13.1) for CP and 0.90 (SECV: 22.3) for oil in g kg −1 on a dry weight basis (second derivative, 400–2500 nm). Prediction models accounted for less than 65, 70 and 72% of the total variation for oil, M and CP, respectively. However, it was concluded that NIRS is a suitable technique to be used as a tool for rapid pre-screening of quality characteristics on breeding programs.

The use of visible and near-infrared reflectance spectroscopy to predict colour on both intact and homogenised pork muscle

Visible and near-infrared reflectance spectroscopy (VIS/NIRS) was used to predict colour on pork muscle samples. Colour values were measured on 44 pork muscles ( n=44) using a Minolta digital camera to determine CIE L* (lightness), CIE a* (redness) and CIE b* (yellowness). Samples were scanned in the visible and near-infrared region of the spectra on a monochromator instrument (400–2500 nm) on both intact and homogenised presentation to the instrument. Two mathematical treatments (first and second derivative) and two scatter transformations to the spectra were explored, none and standard normal variate and detrend (SNV-D). Predictive (VIS/NIRS) calibrations were developed using a modified partial least-squares regression (mPLS) with internal cross validation. The highest coefficients of determination in calibration ( R 2 cal) and lower standard errors in cross validation (SECV) were found for both CIE L* and a*, in homogenised presentation, while poor calibration statistics were found on intact presentation.

The use of near infrared spectroscopy to investigate the composition of silages and the rate and extent of cell-wall degradation

Samples of 25 grass silages and of one maize, one lucerne and one whole crop wheat silage, of known digestibility, chemical composition and cell-wall degradability, were scanned on an near infrared spectrometer (NIRS). Also scanned were samples of the cell-wall (i.e. neutral detergent fibre (NDF)) fraction of the silages and the cell-wall residues produced by incubating the silage in buffered rumen fluid for 3, 8, 16, 24, 45 or 72 h and boiling with neutral detergent solution. The data points were transformed by the standard normal variate and detrending (SNV-D) procedure. Difference spectra were obtained by subtracting the spectrum of the 72 h residue from those of each of the 3, 8, 16, 24 and 45 h residues. The aim was to investigate the extent to which NIRS may be useful as a means of learning more about the digestibility and composition of silages and predicting the rate and extent of cell-wall degradation. The difference spectra of all the silages examined were of similar pattern, with prominent positive regions, indicating components of high degradability at ∼1430–1620 and 1960–2230 nm, a prominent negative region, indicating components of low degradability, at 2230–2400 nm, and other negative regions at 1140–1430 and 1620–1820 nm. Among the 25 grass silages, the spectral regions with the strongest positive correlation with digestibility included those around 1676 and 2228 nm, which corresponded approximately with regions of strong negative correlation with lignin (1672 and 2246 nm) and NDF (1680 and 2214 nm). There were spectral regions of strong positive correlation with fractional rate of cell-wall degradation h −1, indicated by c, around 1684 and 2214 nm, which corresponded with regions of strong negative correlation with silage NDF (1680 and 2214 nm); there were regions of strong negative correlation with c at around 1414 and 1922 nm, which corresponded with regions of strong positive correlation with silage NDF (1406 and 1922 nm). The array of information provided by near infrared (NIR) spectra has the potential to enhance understanding of digestion and cell-wall degradation.

The use of Near Infrared Reflectance Spectroscopy on dried samples to predict biological parameters of grass silage

This study was undertaken to explore the accuracy of Near Infrared Reflectance Spectroscopy (NIRS) for the prediction of in vivo OMD (%) and voluntary intake (g/kg W 0.75) measured through sheep and cattle respectively. A population of 136 grass silages representing a wide range in chemical and biological parameters was used in this investigation. The dried milled silage samples were scanned at 2 nm intervals over the wavelength range 400–2500 nm and the optical data recorded as log 1/Reflectance (log 1/R). The paper examines three multivariate regression techniques: modified partial least squares (MPLS), principal component regression (PCR) and stepwise multiple linear regression (SMLR) and investigates the effect of spectral pretreatment using 1st and 2nd order derivatization with and without three scatter correction procedures: standard normal variate and detrending (SNV-D), normal multiplicative scatter correction (NMSC) and weighted multiplicative scatter correction (WMSC), to optimize accuracy of prediction. The optimum mathematical treatment was selected by minimizing the standard error of prediction (SEP) of a blind validation set using a calibration and validation set of 90 and 46 respectively. The optimum methods were for in vivo organic matter digestibility (OMD), the stepwise regression procedure using 1st derivatization with a scatter correction (SEP 2.4%, R 2 0.87) and for intake the MPLS regression technique again using 1st derivatization and a scatter correction (SEP 4.77 g/kg W 0.75, R 2 0.79). Comparison of three wavelength ranges (1100–2500 nm, 700–2500 nm and 400–2500 nm) on the effect of calibration performance for OMD and intake showed little improvement from extending the range beyond 1100–2500 nm.