SE-PDS enhanced NIR spectral transfer learning: A machine learning approach for cross-instrument jet fuel property quantification.

Removal of external influences from on-line vis-NIR spectra for predicting soil organic carbon using machine learning

Soil salinity monitoring requires accurate measurement of saturated paste extract electrical conductivity (ECe) which is considered the most trustworthy measure of salinity hazard in many laboratories globally. It’s a time-consuming and technically demanding process. In contrast, the measurement of EC values from the 1:1, 1:2.5, and 1:5 soil-to-water ratios is simple, rapid and cheap. This study aims to develop a pedotransfer functions to estimate ECe from diluted soil-to-water extracts (EC1:1, EC1:2.5, EC1:5) and presents an innovative FTIR spectroscopy approach coupled with machine learning for rapid EC prediction. A total of 59 soil samples were collected from 22 profiles across three depths (0–20, 20-50 and 50-100 cm) in Morocco and analyzed for EC at different soil-to-water extracts and scanned using a Bruker-Tensor-II-HTS-XT spectrometer. Random Forest (RF) and Partial Least Squares Regression (PLSR) models were employed to predict ECe and diluted EC values. Results demonstrated a significant linear correlation between the ECe and diluted extract EC values, with an R2>0.89 across all extracts. In addition, the conversion factors (CF) (ECe = CF × EC soil-to-water ratios) varied significantly among soil types, indicating the critical role of soil-type parameters for accurate ECe estimation. FTIR developed models demonstrated high predictive accuracy across all soil-to-water extracts (R2 = 0.86-0.91, RMSE = 0.41-3.69 dS/m), with distinct spectral features at 1970-2550 cm⁻1 and 2867-3086 cm⁻1 identified as the most sensitive regions for EC prediction. Random Forest (RF) models accurately predicted ECₑ from EC1:5 (R2 = 0.92, RMSE = 2.05 dS/m), with enhanced performance when including CEC and CaCO₃ content (R2=0.95, RMSE=1.51 dS/m). In conclusion, FTIR spectroscopy combined with machine learning offers an accurate, rapid and minimal sample preparation method to make it particularly valuable for large-scale precision agriculture. Our findings demonstrate that mid-infrared spectroscopy enabling a rapid ECₑ estimation without saturated paste analysis a significant advancement for salinity hazard monitoring. Graphical Abstract This study aims to develop pedotransfer functions to convert soil electrical conductivity of the saturated paste extract (ECe), a key indicator of soil salinity based on easily measured soil-to-water extract ratios (1:1, 1:2.5, and 1:5) in the irrigated perimeter of the Triffa plain, northeastern of Morocco. In addition, it presents an innovative approach for rapid estimation of the ECe using mid-infrared spectroscopy combined with machine learning techniques. Soil sampling was conducted across three depth intervals and different soil types including Mollisols, Ultisols, Histosols, Entisols and Aridisols. Laboratory analyses included EC measurements from both saturated paste extracts and diluted soil-to-water extracts was conducted. Simultaneously, mid-infrared spectral data were scanned using Fourier-transform infrared (FTIR) spectroscopy. Machine learning algorithms, specifically Random Forest (RF) and Partial Least Squares Regression (PLSR) were used to establish relationships between spectral data and EC values. The study also evaluates the influence of integrating two key soil properties, calcium carbonate (CaCO₃) content and cation exchange capacity (CEC), on model performance and prediction accuracy. The results reveal a strong correlation between the ECe and soil-to-water ratio of 1:1, 1:2.5 and 1:5 with an R2>0,89 across all ratios. In addition, the conversion factors (CF) used to transform soil-to-water ratios into ECe (ECe = CF × EC soil-to-water) varied significantly by soil type for 1:1, 1:2.5 and 1: respectively: Mollisols (1.87, 5.30, 9.52), Ultisols (1.77, 6.03, 8.66), Histosols (2.32, 6.73, 8.74), Aridisols (1.82, 3.76, 7.57), and combined soils (2.09, 5.51, 8.60). Furthermore, the models developed from Mid-infrared spectral data were validated and were recorded as high accuracy for EC1:1 (R2=0.86, RMSE=1.65 dS/m), EC1:2.5 (R2=0.91, RMSE=0.41 dS/m), EC1:5 (R2=0.86, RMSE=0.42 dS/m), and ECe (R2=0.87, RMSE=3.69 dS/m). Moreover, the study reveals that spectral ranges 1970–2550 cm⁻1 and 2867–3086 cm⁻1 were identified as the most sensitive for EC prediction. On the other hand, RF models also demonstrated the strong model performance for ECe prediction from EC1:5 ratio (R2=0.92 and RMSE=2.05 dS/m). Worth noting that integration of soil properties such as CEC and CaCO3, has enhanced the model prediction performances (i.e. R2=0.95 and RMSE=1.51 dS/m). In concluding, FTIR spectroscopy enables accurate prediction of ECe using the (EC) of diluted soil-to-water extracts of 1:1, 1:2.5 and 1:5. Additionally, the RF machine learning algorithms, demonstrated strong potential in estimating ECe from EC1:5, with CEC and CaCO₃ serving a key role in enhancing model performance. Finally, the study recommends that mid FTIR spectroscopy coupled with chemometrics method is a robust, quick and cost-effective method for ECe measurement for soil salinity monitoring.

The objective of this study was to explore the feasibility of near infrared (NIR) spectra calibration model transfer techniques in quantitative analysis of fish meal mixed with soybean meal. One dispersive NIR instrument (Foss 6500) and one Fourier transform (FT) instrument (Nicolet Antaris) were used in this study. The effect of slope/bias correction, local centring, direct standardisation (DS) and piece-wise direct standardisation (PDS) on calibration transfer were studied. When the calibration model based on the Foss 6500 was used directly for the spectra scanned on Nicolet Antaris, it produced unsatisfactory prediction with the RMSEP= 10.81% and bias=-9.73%. However, the predictions were greatly improved after the calibration transfer based on slope/bias correction ( RMSEP=3.19%, bias=0.64.%), local centring ( RMSEP=2.91%, bias=0.50%), DS( RMSEP=3.06%, bias=1.26%) and PDS ( RMSEP=2.54%, bias=0.77%) with the validation set VNI. In order to test the transferability of the four calibration transfer technologies, another independent external validation set (VEI) was used to test the transferability of the four calibration transfer technologies. Similar results were obtained with the prediction ( RMSEP=2.71%, bias=0.64%) for PDS, followed by local centring method ( RMSEP=2.98%, bias=0.52%), DS method ( RMSEP=3.18%, bias=0.96%) and slope/bias correction ( RMSEP=3.24%, bias=0.60%). These findings demonstrated that calibration model transfer technology may be an appropriate tool to quantitatively analyse fish meal mixed with soybean meal.

To correct the spectral changes measured on different instruments in near-infrared (NIR) spectroscopy, a novel hybrid calibration transfer method with nonlinear dimensionality reduction based on direct standardization (DS) and autoencoder (AE), named DS-AE, is reported. First, DS was employed to preliminarily eliminate the spectral difference from the master and slave instruments. Next, the spectral features were extracted by AE to construct the partial least squares (PLS) calibration model. Compared with the linear dimensionality reduction methods, AE learns more latent features of the input spectra that are beneficial to reflect the chemical information of samples. Two NIR experiment datasets, including wheat and corn samples measured on different spectrometers, were employed to evaluate the performance of the DS-AE method. DS, piecewise direct standardization (PDS), canonical correlation analysis (CCA), principal components canonical correlation analysis (PC-CCA), and transfer via an extreme learning machine auto-encoder (TEAM) were introduced for comparative analysis with the proposed method. The results showed that DS-AE provided the lowest root mean squared error of prediction (RMSEP) for the wheat dataset; For the corn dataset, DS-AE provided lower RMSEP than DS, PDS, and CCA, and comparable with PC-CCA and TEAM. The score of the PLS principal components (PCs) describes the spectral differences of different instruments. The results indicated that the hybrid DS-AE method effectively corrected for the spectral variations. In summary, the proposed hybrid DS-AE method provided an alternative for robust standardization of near-infrared spectra measured on different instruments.

Summary We investigated the use of piecewise direct standardization ( PDS ) to remove the effects of water and other environmental factors from proximally sensed (field) visible–near infrared (vis– NIR ) spectra. Our hypothesis was that the PDS ‐standardized field spectra can be used to predict soil carbon effectively with calibrations derived from existing spectroscopic databases of spectra recorded in the laboratory on dried, ground and sieved samples. In our experiments we used field spectra recorded in situ with a portable spectrometer at 124 sites in 11 paddy fields in Z hejiang P rovince, C hina. We sampled the soil at these same sites, recorded their spectra in the laboratory and measured their soil organic carbon ( SOC ) contents with a conventional laboratory technique. Two‐thirds of the samples were used to relate the laboratory spectra to SOC by partial least squares regression ( PLSR ), and the remaining one‐third was used as an independent validation dataset. We selected a representative set of samples from corresponding field and laboratory spectra that we could use as the PDS transfer set. Piecewise direct standardization was used to relate each wavelength in the laboratory spectra to the corresponding wavelength and its neighbours in the field spectra. The field spectra of the validation samples were then corrected with PDS so that they acquired the characteristics of the spectra measured under laboratory conditions. The approach was evaluated by (i) quantifying the similarity between the PDS ‐standardized spectra and their corresponding laboratory spectra, (ii) measuring the accuracy of their SOC predictions on the independent validation dataset and (iii) comparing these results with those of direct standardization ( DS ). Both PDS and DS led to considerable improvements in the predictions of SOC ( R 2 = 0.71, R 2 = 0.60, respectively), compared with those with original field spectra ( R 2 = 0.03). However, fewer transfer samples were needed with PDS to obtain similar results.

To predict the potential geographic distribution of Oncomelania hupensis in Yunnan Province using random forest (RF) and maximum entropy (MaxEnt) models, so as to provide insights into O. hupensis surveillance and control in Yunnan Province. The O. hupensis snail survey data in Yunnan Province from 2015 to 2016 were collected and converted into O. hupensis snail distribution site data. Data of 22 environmental variables in Yunnan Province were collected, including twelve climate variables (annual potential evapotranspiration, annual mean ground surface temperature, annual precipitation, annual mean air pressure, annual mean relative humidity, annual sunshine duration, annual mean air temperature, annual mean wind speed, ≥ 0 ℃ annual accumulated temperature, ≥ 10 ℃ annual accumulated temperature, aridity and index of moisture), eight geographical variables (normalized difference vegetation index, landform type, land use type, altitude, soil type, soil textureclay content, soil texture-sand content and soil texture-silt content) and two population and economic variables (gross domestic product and population). Variables were screened with Pearson correlation test and variance inflation factor (VIF) test. The RF and MaxEnt models and the ensemble model were created using the biomod2 package of the software R 4.2.1, and the potential distribution of O. hupensis snails after 2016 was predicted in Yunnan Province. The predictive effects of models were evaluated through cross-validation and independent tests, and the area under the receiver operating characteristic curve (AUC), true skill statistics (TSS) and Kappa statistics were used for model evaluation. In addition, the importance of environmental variables was analyzed, the contribution of environmental variables output by the models with AUC values of > 0.950 and TSS values of > 0.850 were selected for normalization processing, and the importance percentage of environmental variables was obtained to analyze the importance of environmental variables. Data of 148 O. hupensis snail distribution sites and 15 environmental variables were included in training sets of RF and MaxEnt models, and both RF and MaxEnt models had high predictive performance, with both mean AUC values of > 0.900 and all mean TSS values and Kappa values of > 0.800, and significant differences in the AUC (t = 19.862, P < 0.05), TSS (t = 10.140, P < 0.05) and Kappa values (t = 10.237, P < 0.05) between two models. The AUC, TSS and Kappa values of the ensemble model were 0.996, 0.954 and 0.920, respectively. Independent data verification showed that the AUC, TSS and Kappa values of the RF model and the ensemble model were all 1, which still showed high performance in unknown data modeling, and the MaxEnt model showed poor performance, with TSS and Kappa values of 0 for 24%(24/100) of the modeling results. The modeling results of 79 RF models, 38 MaxEnt models and their ensemble models with AUC values of > 0.950 and TSS values of > 0.850 were included in the evaluation of importance of environmental variables. The importance of annual sunshine duration (SSD) was 32.989%, 37.847% and 46.315% in the RF model, the MaxEnt model and their ensemble model, while the importance of annual mean relative humidity (RHU) was 30.947%, 15.921% and 28.121%, respectively. Important environment variables were concentrated in modeling results of the RF model, dispersed in modeling results of the MaxEnt model, and most concentrated in modeling results of the ensemble model. The potential distribution of O. hupensis snails after 2016 was predicted to be relatively concentrated in Yunnan Province by the RF model and relatively large by the MaxEnt model, and the distribution of O. hupensis snails predicted by the ensemble model was mostly the joint distribution of O. hupensis snails predicted by RF and MaxEnt models. Both RF and MaxEnt models are effective to predict the potential distribution of O. hupensis snails in Yunnan Province, which facilitates targeted O. hupensis snail control.

Spectra transfer based learning for predicting and classifying soil texture with short-ranged Vis-NIRS sensor

Visible near-infrared (VNIR) spectroscopy is a reliable method for estimating soil properties. However, its effectiveness in accurately predicting soil organic carbon (SOC) contents, particularly in wheat-rice rotation fields, remains uncertain. In this study, we collected 202 samples from wheat-rice fields (0–20 ​cm) in southeastern China and measured in-situ spectra of the vertical surface of the soil cores and the laboratory spectra of the dried and sieved soil samples. Our study focused on evaluating three algorithms - external parameter orthogonalization (EPO), direct standardization (DS), and piecewise direct standardization (PDS) - to address the influence of external factors, particularly soil moisture. To carry out our analysis, the dataset was divided into calibration (141 samples) and validation (61 samples) sets via the Kennard-Stone algorithm. A subset of the corresponding in-situ and laboratory spectra in the calibration set (transfer set) was used to derive the transfer matrix for EPO, DS, and PDS, enabling the conversion of in-situ spectra to laboratory spectra by characterizing their differences. Four machine learning models, including cubist, partial least squares regression (PLSR), random forest (RF), and memory-based learning (MBL), were used to predict the SOC, particulate organic carbon (POC), and mineral-associated organic carbon (MAOC) contents based on the laboratory, in-situ, and corrected in-situ spectra. The results revealed that the laboratory spectra outperformed the non-corrected in-situ spectra, with coefficients of determination (R2) of 0.91, 0.75, and 0.80 for SOC, POC, and MAOC, respectively. Among the models, MBL and PLSR exhibited the highest average R2 at 0.85–0.86. EPO marginally improved the prediction accuracy (R2 increased from 0.85 to 0.87 for SOC, 0.64 to 0.69 for POC, and 0.75 to 0.82 for MAOC). These promising prediction accuracies underscore the potential of VNIR spectra for in-situ predictions in wheat-rice fields in Southeast China, offering insights for predicting SOC contents via in-situ spectroscopy.

In order to reduce the time and cost for near infrared (NIR) model development and maintenance, the transfer of NIR spectra measured on four different portable spectrometers (one master and three target instruments) for predicting the lignin content of pulp wood is investigated in this work. Eighty-two wood samples were prepared by chipping and grinding, and their NIR spectra were recorded with four spectrometers. Calibration models for the determination of lignin in pulp wood have been developed by partial least squares (PLS) regression, while average Mahalanobis distances (AMD) and average differences of spectra (ADS) were used to quantify the spectral differences. Then piecewise direct standardization (PDS) has been applied, and compared to direct standardization (DS), slope/bias correction (SBC) and canonical correlation analysis (CCA). The accuracy of the models has been evaluated by comparing their prediction performance. The results indicated that the prediction performances of the three target instruments are greatly improved by using the three algorithms. The advantage of the PDS algorithm is that fewer samples are required for the transfer sets, which means lower model maintenance cost for practical applications. When it comes to window size setting procedure, it was found that if there are large spectral differences between the master and the target spectrometer, a large window size should be used and if the spectral difference is a significant lateral shift, an asymmetric window with appropriate window size is necessary to ensure a good transfer performance for the PDS algorithm.

By employing an Acousto-optic Tunable Filter (AOTF) as the spectroscopical device, a Near-infrared (NIR) spectrometer that is compact, rugged and having random wavelength access has been built. In this paper, the working principle of Acousto-optic Tunable Filter and the schematic design of the instrument are described in details. As a spectrometer for on-site use, the instrument adopts a double beam scheme for self-calibration, modulation of light intensity is employed to reduce interference of noise, and different configurations of the probe offer more versatility in measurement. Specifications for the instrument are quantified. After the instrument’s performance is qualified for wavelength accuracy and scale precision, glucose solution samples are prepared and transflectance spectra are sampled on the instruments. PLS model has been set up through the spectra of aqueous solution of glucose and cross-validation is used to test its predictability. Calibration transfer is attempted between instruments. Direct Standardization (DS) and Piecewise Direct Standardization (PDS) algorithms are briefly described here. It should be noted in this experiment that the proper determination of rank of transformation matrix plays an important role in estimating the quality of spectra transfer. Calibration transfer is performed between instruments and a satisfactory prediction error of 1.5~2 times larger is achieved by applying DS and PDS method.

HighlightsThe RC-PLSR model for Haiwan scallop can be transferred to Xiayi scallop.The direct standardization method is suggested for model maintenance.The VSWS-PDS method can be further improved in precision.Abstract. A prediction model for evaluating the moisture content in dried Haiwan scallops was established using hyperspectral imaging (HSI) technology in a previously published study. The accuracy of such models is usually affected by differences in sample species, different environmental conditions such as temperature or humidity, and aging of instruments. In this study, the prediction ability of the RC-PLSR model is improved by correcting the spectra of the tested species of dried scallop (i.e., Xiayi) to solve the problem of model failure caused by sample differences. The results of model maintenance by direct standardization (DS) are compared with those of variety sensitive wavelength selection - piecewise direct standardization (VSWS-PDS). The results showed that after using VSWS-PDS to modify the spectral data of the dried scallop samples, the correlation coefficient of prediction (Rp) of the updated model increased from 0.0890 to 0.9190. However, the root mean square error of prediction (RMSEP) also increased, indicating a need for improved precision. The RC-PLSR model based on DS correction showed Rp of 0.790 and RMSEP of 9.7481%. Model maintenance using the DS method is suggested because DS generally outperformed VSWS-PDS, even with a lower correlation coefficient. Future work on error reduction and sample input is suggested for VSWS-PDS optimization. Keywords: Direct standardization, Hyperspectral images, Model maintenance, Scallop, VSWS-PDS.

Calibration transfer of partial least squares jet fuel property models using a segmented virtual standards slope-bias correction method

Calibration transfer of a Raman spectroscopic quantification method for the assessment of liquid detergent compositions between two at-line instruments installed at two liquid detergent production plants

Low-field proton nuclear magnetic resonance (LF-1H NMR) devices based on permanent magnets are a promising analytical tool to be extensively applied to the process analytical chemistry scenario. To enhance its analytical applicability in samples where the spectral resolution is compromised, multivariate regression methods are required. However, building a robust calibration model, such as partial least squares (PLS) regression, is a laborious task because (1) the number of measurements required during the calibration process is large and (2) the procedure must be repeated when the instrument is changed or after a certain period due to the long-term stability of the instrument. Thus, the present work describes the application of calibration transfer methodologies (direct standardization (DS), piece-wise direct standardization (PDS), and double-window piece-wise direct standardization (DWPDS)) on LF-1H NMR to exempt the necessity of a recalibration procedure when moving from the original spectrometer to a second one with the same, lower, or higher magnetic field. These calibration transfer methodologies were tested with PLS models built on a 60 MHz (for the proton Larmor frequency) spectrometer to predict the specific gravity (SG), distillation temperature (T50%), and final boiling point (FBP) of commercial gasoline. The results showed that the DWPDS method applying only 2 to 7 transference samples enables the transference of all PLS models built on the primary instrument (60 MHz) to other (43, 60, and 80 MHz) different instruments, reaching the same RMSEP values as the primary instrument: 1.2 kg/m3 for SG, 5.1 °C for FBP, and 1.1 °C for T50%.

Predicting soil organic carbon content using simulated insitu spectra and moisture correction algorithms in southern Xinjiang, China