Near-infrared Spectral Datasets Research Articles

Flexible optimization of variables based on exponential and linear attenuation elimination-binary dragonfly algorithm in near infrared spectroscopic analysis

Single Binary Dragonfly Algorithm (Single-BDA) which is an intelligent optimization algorithm, normally needs more calculation time and often obtains unrobust variables. In this study, an Exponential and Linear Attenuation Elimination-Binary Dragonfly Algorithm (ELAE-BDA) is proposed to overcome the problems by combining exponential and linear attenuation functions. The algorithm can quickly eliminate a part of the variables with large RMSECV in the iterative process through the exponential attenuation function first, and use the linear attenuation function at the end of the iteration for meticulous elimination of invalid variables. It avoids the important variables being deleted unexpectedly, greatly improves the running speed of the algorithm, reduces the randomness of results, and significantly improves the analysis ability of the model. Three near-infrared spectral datasets are used as research objects to evaluate the performance of the ELAE-BDA algorithm, and 6 wavelength selection algorithms including Single-BDA, GA, CARS, MC-UVE, SCARS, and fiPLS are used for comparison. The results show that the ELAE-BDA algorithm can obtain results with almost the smallest number of wavelengths and the highest stability. The established PLSR models have the best prediction with the lowest RMSECV and RMSEP, indicating that the proposed algorithm can provide a more accurate and effective results of wavelength optimization for near-infrared spectral analysis.

Comparison of ATR-FTIR and NIR spectroscopy for identification of microplastics in biosolids

Biosolids are considered a potentially major input of microplastics (MPs) to agricultural soils. Our study aims to identify the polymeric origin of MPs extracted from biosolid samples by comparing their Attenuated Total Reflection (ATR) - Fourier-transform infrared (FTIR) spectra with the corresponding near-infrared (NIR) spectra. The reflectance spectra were preprocessed by Savitzky-Golay (SG), first derivative (FD) and compared with analogous spectra acquired on a set of fifty-two selected commercial plastic (SCP) materials collected from readily available products. According to the results portrayed in radar chart and built from both ATR-FTIR and NIR spectral datasets, the MPs showed high correlations with polymers such as polyethylene (LDPE, HDPE), polyethylene terephthalate (PET), polystyrene (PS), polypropylene (PP) and polyamide (PA), determined in SCP samples. Each unknown MP sample had on average three or more links to several types of SCP, according to the correlation coefficients for each polymer ranging from 0.7 up to 1. The comparison analysis classified the majority of MPs as composed mainly by LDPE/HDPE, according to the top correlation coefficients (r > 0.90). PP and PET were better identified with NIR than ATR-FTIR. In contrast to ATR-FTIR analysis, NIR was unable to identify PS. Based on these results, the primary sources of MPs in the biosolids could be identified as discarded consumer packaging (containers, bags, bottles) and fibers from laundry, disposable glove, and cleaning cloth. SynopsisMicroplastics (MPs) are considered contaminants of emerging concern. This study compares two simple and fast spectroscopy techniques to identify microplastics in the biosolid matrix.

MAE-NIR: A masked autoencoder that enhances near-infrared spectral data to predict soil properties

Soil available nutrients are crucial for promoting crop growth, and controlling their content is essential for increasing yield, promoting smart agriculture, and protecting the environment. Near-infrared spectroscopy technology enables the efficient and nondestructive detection of soil nutrient content in real time. However, current near-infrared spectroscopy datasets suffer from data isolation due to the inability to share data feature advantages, necessitating expensive data acquisition. To address this issue, we propose an unsupervised method for near-infrared spectral enhancement to analyse soil samples from the red soil of southern Anhui. Our proposed framework, named MAE-NIR, is a near-infrared spectral masked autoencoder that learns highly robust and generic spectral features from abundantly available public near-infrared spectral datasets. We collected near-infrared spectroscopy data from a depth of 900–1700 nm and utilized the publicly available spectral dataset LUCAS 2009 to reconstruct the spectral waveform details. This method facilitates the capture of both local and global aspects, thereby aiding subsequent downstream tasks. Several renowned regressors, such as partial least squares regression, random forest, and neural networks, are also employed to assess the effectiveness of near-infrared spectral enhancement using MAE-NIR. The spectral enhancement method based on the masked autoencoder significantly outperforms all other spectral preprocessing methods. The coefficient of determination (R2) values of the best models of available nitrogen, phosphorus, and potassium in the soil increased to 0.941, 0.926, and 0.903, respectively, which are on average 22.42%, 11.14%, and 10.35% higher than those obtained from the previous best preprocessing methods. This indicates the efficacy of using MAE-NIR to predict soil nutrients, as it effectively enhances the accuracy of nutrient content measurements.

Application of improved Stacking ensemble learning in NIR spectral modeling of corn seed germination rate

Stacking ensemble learning is one of the most effective integration technologies and is increasingly applied to near-infrared spectroscopy combined with chemometrics methods. The prediction accuracy of Stacking is primarily affected by the selection of different models. However, many current studies are mainly artificial selection models' combinations. It affects the model's prediction accuracy and increases the algorithm's difficulty. It is difficult to efficiently and accurately find the optimal configuration scheme. This study applies a genetic algorithm to find the optimal base and meta learner combinations in Stacking ensemble learning. This method uses the near-infrared spectral data set of corn seed germination rate. First, select the best pretreatment methods for seven models, including Gaussian process regression (GPR), SVR, PLS, etc. The above seven single learners after pretreatment are taken as the candidate base learner, and then random forest (RF), SVR, PLS, and GPR are taken as the potential meta learner; use a genetic algorithm to select the optimal model combination configuration and generate GA-Stacking algorithm. The model prediction results of the improved model GA-Stacking are compared with several single models and Stacking ensemble learning via the artificial selection model combinations. The results show that the prediction performance using the GA-Stacking ensemble learning model is optimal, R2 is 0.9022, and RMSE is 0.1100. The experiment shows that the model combination selected by the genetic algorithm has significantly improved the prediction performance of the Stacking ensemble learning model and reduced the risk of the model's overfitting.

A guide for quality control of honey: Application of UV–vis scanning spectrophotometry and NIR spectroscopy for determination of chemical profiles of floral honey produced in southern Brazil

This study aimed to determine the chemical profile of floral honey by ultraviolet and visible spectrophotometry (UV–vis) and near-infrared (NIR) vibrational spectroscopy, and via analysis of the total phenolic content, considering the harvest year (2018–2019 and 2019–2020), geographic and botanical origins, and seasonality of samples (n = 116) collected in southern Brazil. The total phenolic content (TPC) of honey showed great variation by geographic origin and for both harvests, with a higher overall average in the 2019–2020 (3.81–7.22 mg/mL) compared to the 2018–2019 (2.49–6.17 mg/mL), which seemed to be related to climatic factors, mainly precipitation. The visual inspection of the UV–vis spectra (280–320 nm) revealed two samples with distinct chemical compositions, being these further confirmed as outlier by PCA. UV–vis (260–360 nm) coupled to PCA was able to distinguish honey with higher and lower TPC. The NIR spectral data set revealed discrepancies among honey according to the harvest seasons, with bands associated with higher water content and lower carbohydrate amounts in the 2018–2019-collected samples; the contrary has been noted for the 2019–2020. Thus, UV–vis and NIR spectroscopies coupled to PCA proved to be a useful approach to distinguish honey samples as to possible adulterations and geographic origin associated with their higher and lower phenolic contents.

Semi-Supervised Deep Learning-Based Multi-component Spectral Calibration Modeling for UV-vis and Near-Infrared Spectroscopy without Information Loss.

Spectral analysis is an important method for characterizing and identifying chemical species. However, quantitative spectral analysis of multiple chemical properties in the real world has always been a challenging problem due to the strong correlation, massive noise, and serious information overlapping of the spectral features. Here, we present a new semi-supervised spectral calibration method based on information lossless decoupling of spectral features named NICEM. To realize the separation and extraction of key latent features, the method uses the flow-based model non-linear independent component estimation (NICE) to learn the sample distribution. The spectral data information is transformed into independent latent variables obeying Gaussian distribution by the reversible structure of deep network without information loss, so as to find the essential properties and realize the feature nonlinear decomposition. Moreover, the association between the input latent feature variables and attributes is evaluated by the maximum mutual information coefficient to eliminate the adverse effects of irrelevant information in the latent variable space and mine key information. Since the latent variables are independent in each dimension, the NICEM method is easier to establish an accurate semi-supervised multi-component calibration model even for high overlapping and complex spectral data. The applicability of the proposed spectral modeling method is demonstrated by using three ultraviolet-visible and near-infrared spectral data sets with 15 physical and chemical properties including diesel fuels, corn, and multi-metal ions solution. Results show that the proposed NICEM method has the highest determination coefficient (R2) and significantly improves extrapolation compared with the seven state-of-the-art methods. The proposed method is intuitive because it obviates complex feature engineering and prior knowledge and is a promising spectral calibration tool for quantitative analysis in other spectroscopy applications.

A new near-infrared spectral transformation method for quantitative analysis

The existing near-infrared spectrum preprocessing methods are limited in improving the performance of prediction models for quantitative analysis because they are hard to extract furthermore valid information from the complex near-infrared spectra. This paper proposes a new near-infrared spectral transformation method based on the Gaussian-curve-fitting—adaptive full-rank fitting integral (AFFI) transformation, which can automatically adapt to different spectral shapes and fully display the useful information hidden in the waveforms. In this method, the optimal bandwidths of the Gaussian functions are determined by correlation analysis between waveform and bandwidths; the heights of the Gaussian functions are obtained by constructing and solving the curve-fitting equations; the transformation is realized by integrating these Gaussian functions. In the experiments, the variation of the Pearson correlation coefficient between the near-infrared spectral bands and quantitative analysis object is used to demonstrate the effectiveness of the proposed method. Finally, the partial least squares (PLS) method was used to build prediction models of quantitative analysis. The experimental results of two different near-infrared spectral data sets show that the proposed method has obvious advantages with some existing pretreatment and transformation methods.

An automatic generation of pre-processing strategy combined with machine learning multivariate analysis for NIR spectral data

Pre-processing near-infrared (NIR) spectral data is indispensable in multivariate analysis, since the measured spectra of complex samples are often subject to overwhelming background, light scattering, varying noises, and other unexpected factors. Various pre-processing methods have been developed to remove or reduce the interference of these effects. Until now, most applications of NIR spectra pre-processing in multivariate calibration have been trial-and-error, with selecting a proper method depending on the nature of the data, expertise, and practitioner experience. Thus, it is usually challenging to determine the best pre-processing method for a given data. In order to tackle these problems, this study proposes a new concept of data pre-processing, namely, automatically generating a pre-processing strategy (AGoES). This concept belongs to the ensemble pre-processing method, where machine learning algorithms (PLSR, SVM, k-NN, DT, AB, and GPR) built on differently preprocessed data are combined by 5-fold cross-validation and grid search optimization. To investigate our concept, a public NIR spectral dataset was used to predict three responses, including dry matter content (DM), organic matter content (OM) and ammonium nitrogen content (AN) from manure organic waste. The results show that SVM is the best algorithm combined with the AGoES pre-processing to predict DM and AN with a ratio of prediction to deviation (RPD) of 3.619 and 2.996, respectively. The AB tandem with AGoES pre-processing is the best strategy for predicting OM with an RPD of 3.185. Therefore, in the framework of the AGoES concept, it is unsupervised pre-processing, more simple, and feasible to apply multivariate analysis using machine learning algorithms.

Predictions of multiple food quality parameters using near-infrared spectroscopy with a novel multi-task genetic programming approach

In order to meet the increasing demand for food safety and quality, new methods for simultaneous and rapid determination of multiple food quality parameters (FQPs) are urgently needed in the food industry. Incorporating near-infrared (NIR) spectroscopy and spectral prediction model for rapid, repeatable, non-destructive, and low running costs quantitative analysis of FQPs is enjoying increasing popularity in the food industry. However, most existing spectrum-based prediction models are trained under a single-task learning framework, that is, a prediction model for each quality parameter and spectrum is constructed separately. This paradigm ignores possible connections among prediction tasks of different FPQs, which may result in the performance degradation of a single FPQ prediction model. This study proposes a novel multi-task genetic programming-based approach named EM4GPO for building multiple FQPs predictions simultaneously. In EM4GPO, the multi-dimensional trees are used to encode the raw NIR spectrum to shared features of multiple FQPs; for each FQP, a least square support vector regression (LS-SVR) modeling is performed on the shared features to obtain private features and prediction model; during the optimization process, a new algorithm is developed to optimize the previously obtained shared and private features, and LS-SVR prediction models through population evolution by combining the multidimensional multiclass genetic programming with multidimensional populations optimization method with nondominated sorting method. The proposed EM4GPO model is evaluated and compared with nine popular NIR prediction models using 10 NIR spectral datasets. The experimental results showed that EM4GPO outperformed other commonly used methods in all datasets which indicates that EM4GPO is competitive and effective in solving the problem of multiple FQPs predictions using the NIR spectrum.

Pre-processing ensembles with response oriented sequential alternation calibration (PROSAC): A step towards ending the pre-processing search and optimization quest for near-infrared spectral modelling

Ensemble pre-processing is emerging as a potential tool to avoid the tiring pre-processing selection and optimization task in near-infrared (NIR) spectral modelling. Furthermore, differently pre-processed data may carry complementary information, hence, ensemble pre-processing may represent the best suited modelling option to extract all the useful information from differently pre-processed data. Recently, multi-block techniques such as sequential (SPORT) and parallel (PORTO) orthogonalized partial least squares regression were proposed to extract complementary information present in differently pre-processed data. Although such multi-block techniques allowed efficient modelling of differently pre-processed data blocks, depending on the approach, challenges related to choosing block order, parameter tuning, block scaling and optimization time requirements still must be dealt with. To cope with such issues, the present study proposes the use of a recently developed faster, block order independent and scale independent, multi-block data modelling technique called response-oriented sequential alternation (ROSA) to process the multi-block data generated by differently pre-processing the same NIR data. This new method is called PROSAC, i.e., pre-processing ensembles with ROSA calibration. The potential of the approach is demonstrated on five real NIR spectral datasets. Furthermore, as baselines for comparison, partial least squares regression was done on individually pre-processed data sets, and using two multi-block pre-processing fusion approaches, i.e., SPORT and PORTO. The ensemble pre-processing with ROSA achieved either better performance compared to the baseline methods or achieved comparable performance without the need to worry about the pre-processing order, the scaling of data after pre-processing and optimization time requirements. PROSAC can be considered as a general tool for the ensemble pre-processing for NIR data modelling. • The method was compared with recent novel multi-block pre-processing ensemble tools. • The method is fast, order and block scale independent thus complements pre-processing ensembles. • The method was evaluated on five NIR spectroscopy datasets. • The method eliminates the need for pre-processing selection.

Sparse Reconstruction Using Block Sparse Bayesian Learning With Fast Marginalized Likelihood Maximization for Near-Infrared Spectroscopy

The absorption peak contains a great amount of important chemical information that is critical for the qualitative/quantitative analysis of organic compounds in high-dimensional near-infrared (NIR) spectral data. Sparse Bayesian learning (SBL) algorithm can capture key information from the underlying signal with only a small number of measurements. The computational cost is high because the inverse of a large matrix must be calculated in each iteration with the use of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{1}$ </tex-math></inline-formula> -norm approximation or least squares. To overcome this problem, an improved block SBL (BSBL) is adopted to first reconstruct the absorption peak variables by determining the position and width of the limited peaks autonomously. Then, the fast-marginalized likelihood maximization (FMLM) is used to obtain the optimal hyperparameters of the cost function and realize a fast convergence rate. The proposed method is verified using two NIR spectral datasets and compared with the full-spectral-based partial least squares (PLS) and the other three models. The results demonstrate that the proposed method has a low computational cost without losing prediction performance for the NIR spectroscopy (NIRS).

Prediction of various soil properties for a national spatial dataset of Scottish soils based on four different chemometric approaches: A comparison of near infrared and mid-infrared spectroscopy

Infrared spectroscopic techniques, in combination with chemometric approaches, have been widely used to estimate different physical and chemical properties in soil samples. This study aims to assess the performance of diffuse reflectance spectroscopy in the near-infrared (NIR) region and Attenuated Total Reflection Fourier Transform infrared spectroscopy (ATR-FTIR) in the mid-infrared (MIR) region to predict nine different soil properties: total carbon, total nitrogen, bulk density, clay, sand, silt, pH (in H2O), exchangeable Mg and exchangeable K using chemometric approaches. The predictive performance of four different regression methods i.e., Partial least square (PLS), support vector regression (SVR), Cubist and convolutional neural network (CNN) in combination with different pre-processing approaches were investigated. For CNN, the FTIR/NIR spectra were converted to spectrograms and fed into the CNN to examine the prediction accuracy of two-dimensional convolutional neural network (CNN-2D) and to compare its performance with one-dimensional convolutional neural networks (CNN-1D, spectral data as input). To achieve these objectives, we used a spectral library of 650 samples from National Soils Inventory of Scotland (NSIS) dataset collected in a 20 km grid throughout Scotland between 2007 and 2009. The FTIR and NIR data were both split into calibration (nc = 520) and validation (nv = 130) sets. Our results show that the regression models with FTIR data have better predictive performance than those created using NIR data for all the studied soil properties (improvement in root mean square error of prediction (RMSEP) of 3–61%), except for pH. Comparing the different chemometric approaches, for both the NIR and FTIR, the results indicated that the CNN-1D models performs better than PLS, SVR and CNN-2D for all the studied soil components in terms of RMSEP. We found that the CNN-1D models created using the NIR spectral dataset were superior (lower RMSEP values) to those developed using the Cubist approach for total carbon, total nitrogen, clay, silt, pH and exchangeable K, whereas the Cubist models for sand and exchangeable Mg performed slightly better than CNN-1D. For the FTIR data, Cubist models for total carbon and silt performed best (with lowest RMSEP and highest Residual Prediction Deviation (RPD) values), while for the rest of the components, CNN-1D outperformed the PLS, SVR, Cubist and CNN-2D. We also calculated the impact of the NIR and FTIR variables used by the Cubist and CNN-1D models in predicting different soil properties. We used the Shapley Additive Explanation (SHAP) values, a game theoretic approach, to interpret the output of the CNN-1D models.

Wheat Kernel Variety Identification Based on a Large Near-Infrared Spectral Dataset and a Novel Deep Learning-Based Feature Selection Method.

Near-infrared (NIR) hyperspectroscopy becomes an emerging nondestructive sensing technology for inspection of crop seeds. A large spectral dataset of more than 140,000 wheat kernels in 30 varieties was prepared for classification. Feature selection is a critical segment in large spectral data analysis. A novel convolutional neural network-based feature selector (CNN-FS) was proposed to screen out deeply target-related spectral channels. A convolutional neural network with attention (CNN-ATT) framework was designed for one-dimension data classification. Popular machine learning models including support vector machine (SVM) and partial least square discrimination analysis were used as the benchmark classifiers. Features selected by conventional feature selection algorithms were considered for comparison. Results showed that the designed CNN-ATT produced a higher performance than the compared classifier. The proposed CNN-FS found a subset of features, which made a better representation of raw dataset than conventional selectors did. The CNN-ATT achieved an accuracy of 93.01% using the full spectra and keep its high precision (90.20%) by training on the 60-channel features obtained via the CNN-FS method. The proposed methods have great potential for handling the analyzing tasks on other large spectral datasets. The proposed feature selection structure can be extended to design other new model-based selectors. The combination of NIR hyperspectroscopic technology and the proposed models has great potential for automatic nondestructive classification of single wheat kernels.

Photometric Normalization of Chang’e-4 Visible and Near-Infrared Imaging Spectrometer Datasets: A Combined Study of In-Situ and Laboratory Spectral Measurements

Until 29 May 2020, the Visible and Near-Infrared Imaging Spectrometer (VNIS) onboard the Yutu-2 Rover of the Chang’e-4 (CE-4) has acquired 96 high-resolution surface in-situ imaging spectra. These spectra were acquired under different illumination conditions, thus photometric normalization should be conducted to correct the introduced albedo differences before deriving the quantitative mineralogy for accurate geologic interpretations. In this study, a Lommel–Seeliger (LS) model and Hapke radiative transfer (Hapke) model were used and empirical phase functions of the LS model were derived. The values of these derived phase functions exhibit declining trends with the increase in phase angles and the opposition effect and phase reddening effect were observed. Then, we discovered from in-situ and laboratory measurements that the shadows caused by surface roughness have significant impacts on reflectance spectra and proper corrections were introduced. The validations of different phase functions showed that the maximum discrepancy at 1500 nm of spectra corrected by the LS model was less (~3.7%) than that by the Hapke model (~7.4%). This is the first time that empirical phase functions have been derived for a wavelength from 450 to 2395 nm using in-situ visible and near-infrared spectral datasets. Generally, photometrically normalized spectra exhibit smaller spectral slopes, lower FeO contents and larger optical maturity parameter (OMAT) than spectra without correction. In addition, the band centers of the 1 and 2 μm absorption features of spectra after photometric normalization exhibit a more concentrated distribution, indicating the compositional homogeneity of soils at the CE-4 landing site.

Domain adaptive partial least squares regression

In practical applications, the problem of training- and test-samples from different distributions is often encountered, such as instruments or external environmental factors change when measuring the data. Therefore, a multivariate calibration model established, based on the training set needs to be adaptive to meet the requirements of test samples from different domains. Extracting domain adaptive latent variables is an effective method to address such issues, but related studies about making multivariate calibration models adaptive to different domains in unsupervised methods are rarely reported. In this paper, a domain adaptive partial least squares regression is proposed, which uses the Hilbert-Schmidt independence criterion to evaluate the independence of the extracted latent variables and domain labels. In both the original space and reproducing kernel Hilbert space, the proposed method can obtain a closed form of the domain adaptive projection vector, which ensures the high efficiency of the model calculation. The validity of the proposed method is verified by several simulation data sets and real near-infrared spectral data sets. The experimental results show that the proposed method has good universality for unsupervised regression calibration transfer.

Compression strength prediction of Xylosma racemosum using a transfer learning system based on near-infrared spectral data

A transfer learning system was designed to predict Xylosma racemosum compression strength. Near-infrared (NIR) spectral data for Acer mono and its compression strength values were used to resolve the weak generalization problem caused by using a X. racemosum dataset alone. Transfer component analysis and principal component analysis are domain adaption and feature extraction processes to enable the use of A. mono NIR spectral data to design the transfer learning system. A five-layer neural network relevant to the X. racemosum dataset, was fine-tuned using the A. mono dataset. There were 109 A. mono samples used as the source dataset and 79 X. racemosum samples as the target dataset. When the ratio of the training set to the test set was 1:9, the correlation coefficient was 0.88, and mean square error was 8.84. The results show that NIR spectral data of hardwood species are related. Predicting the mechanical strength of hardwood species using multi-species NIR spectral datasets will improve the generalization ability of the model and increase accuracy.

Calibration Transfer Based on Affine Invariance for NIR without Transfer Standards.

Calibration transfer is an important field for near-infrared (NIR) spectroscopy in practical applications. However, most transfer methods are constructed with standard samples, which are expensive and difficult to obtain. Taking this problem into account, this paper proposes a calibration transfer method based on affine invariance without transfer standards (CTAI). Our method can be utilized to adjust the difference between two instruments by affine transformation. CTAI firstly establishes a partial least squares (PLS) model of the master instrument to obtain score matrices and predicted values of the two instruments, and then the regression coefficients between each of the score vectors and predicted values are computed for the master instrument and the slave instrument, respectively. Next, angles and biases are calculated between the regression coefficients of the master instrument and the corresponding regression coefficients of the slave instrument, respectively. Finally, by introducing affine transformation, new samples are predicted based on the obtained angles and biases. A comparative study between CTAI and the other five methods was conducted, and the performances of these algorithms were tested with two NIR spectral datasets. The obtained experimental results show clearly that, in general CTAI is more robust and can also achieve the best Root Mean Square Error of test sets (RMSEPs). In addition, the results of statistical difference with the Wilcoxon signed rank test show that CTAI is generally better than the others, and at least statistically the same.

A two-level strategy for standardization of near infrared spectra by multi-level simultaneous component analysis

Standardization of near infrared (NIR) spectra is indispensable in practical applications because the spectra measured on different instruments are commonly used and the difference between the instruments must be corrected. A two-level standardization method is proposed in this study based on multi-level simultaneous component analysis (MSCA) algorithm for correcting the spectral difference between instruments. A two-level MSCA model is used to model the difference between instruments (the first level) and samples (the second level). With the two models, the spectral difference due to instruments and measurement operation can be corrected, respectively. Three NIR spectral datasets of pharmaceutical tablet, corn and plant leaf are used to evaluate the efficiency of the proposed method. The results show that the score of the first level model describes the overall spectral difference between instruments, and the score of the second level model depictures the spectral difference of the same sample between the measurements. The latter difference may include the spectral variations caused by instrument, operation and the measurement conditions. Therefore, both the spectral difference due to the instrument and measurement can be corrected by adjusting the coefficients in the scores of the two level models, respectively. The proposed method provides a good way for standardizing the spectra measured on different instruments when the measurement is not reproducible.

A novel multivariate calibration method based on variable adaptive boosting partial least squares algorithm

With the aid of chemometric methods, near-infrared (NIR) spectroscopy is widely applied to the analysis of complex samples. Boosting partial least squares (BPLS) is one of the important approaches and has many applications to improve the predictive stability and accuracy of the NIR models. However, accurate calculation results are difficult to obtain due to the redundant variables that contribute more collinearity and noise than relevant information to models. In this work, an algorithm named as variable adaptive boosting partial least squares (VABPLS) was proposed to get higher robustness models and enhance the prediction ability. The theory and calculation of VABPLS are just similar with BPLS algorithm, but a variable adaptive strategy based on adaptive reweighted sampling (ARS) theory is fused into the algorithm to improve the accuracy, instead of simply adding. Simultaneous weighting of samples and variables in the boosting series is found to be more effective than the single weighting. The performance of VABPLS is tested with the NIR spectral datasets of corn and tobacco leaf samples. Results show that the advantages of VABPLS are many, such as enhanced accuracy, less parameters and easy to implement, compared with the traditional approaches.

Variable space boosting partial least squares for multivariate calibration of near-infrared spectroscopy

A novel boosting strategy by establishing sub-model from variable direction named variable space boosting partial least squares (VS-BPLS) was proposed for multivariate calibration of near-infrared (NIR) spectroscopy. At the first cycle, all the variables in the training set are given the same sampling weights and then a certain number of variables are selected to build PLS sub-model according to the distribution of the sampling weights. In the following cycles, the sampling weights of the variables in the training set are given by a predefined loss function. This loss function is about the error of known and predicted spectra that is obtained by the product of score and loading of PLS sub-models. The final prediction for unknown sample is obtained by the weighted average of each prediction of all the sub-models. The proposed method not only can solve the small sample problem, but also remove redundant information in variables. The performance of VS-BPLS is tested with two NIR spectral datasets. As comparisons to VS-BPLS, the conventional PLS and two variable selection method Monte Carlo uninformative variable elimination PLS (MCUVE-PLS) and randomization test PLS (RT-PLS) have also been investigated. Results show that VS-BPLS has a superiority for small sample problems in prediction accuracy and stability compared with the PLS, MCUVE-PLS and RT-PLS.