Spectral Data Reduction Research Articles

Application of attenuated total reflectance—Fourier transform infrared spectroscopy—(ATR-FTIR) and principal component analysis (PCA) in identification of copying pencils on different paper substrates

Presence of copying pencils in heritage objects poses a significant challenge for conservators due to their proneness to fading, sensitivity to solvents and difficulties in differentiation from regular graphite pencils. In this paper a method of copying pencils identification by means of ATR—FTIR spectroscopy is used. A protocol for spectra processing and dimensionality reduction of spectral data by means of principal component analysis has been developed, allowing for pencil types differentiation and providing an easy to read visual representation of the collected data. The protocol has been developed on mock-up samples and tested on objects from the archives of the State Museum Auschwitz – Birkenau.

Machine Learning and Feature Selection for soil spectroscopy. An evaluation of Random Forest wrappers to predict soil organic matter, clay, and carbonates

Soil spectroscopy estimates soil properties using the absorption features in soil spectra. However, modelling soil properties with soil spectroscopy is challenging due to the high dimensionality of spectral data. Feature Selection wrapper methods are promising approaches to reduce the dimensionality but are barely used in soil spectroscopy. The aim of this study is to evaluate the performance of two feature selection wrapper methods, Sequential Forward Selection (SFS) and Sequential Flotant Forward Selection (SFFS) built using the Random Forest (RF) algorithm, for dimensionality reduction of spectral data and predictive modelling of modelling soil organic matter (SOM), clay and carbonates. The reflectance of 100 soil samples, acquired from Sierra de las Nieves (Spain), was measured under laboratory conditions using ASD FieldSpec Pro JR. Four different datasets were obtained after applying two spectral preprocessing methods to raw spectra: raw spectra, Continuum Removal (CR), Multiplicative Scatter Correction (MSC), and a so-called “Global” dataset composed of raw, CR and MSC features. The performance of RF models built with feature selection methods was compared to that of Partial Least Squares Regression (PLSR) and RF (alone).RF models built with SFS and SFFS outperformed PLSR and RF alone models: The best RF models with feature selection had a respective ratio of performance to interquartile distance of 1.93, 0.38 and 2.56. PLSR models had an accuracy of 1.41, 0.29 and 1.81 for SOM, carbonates, and clay, respectively. RF alone had a respective performance of 1.29, 0.29 and 1.81. The application of feature selection wrapper methods reduced the number of features to less than 1 % of the starting features. Features were selected across all spectra for SOM and clay, and around 900 nm, 1900 nm, and 2350 nm for carbonates. However, feature selection highlighted features around 1100 nm in SOM modelling, as well as other features around 2200 nm, which is considered a main absorption feature of clay. The application of feature selection with Random Forest was very important in improving modelling accuracy, reducing the redundant features and avoiding the curse of dimensionality or Hughes effect. Thus, this research showed an alternative to dimensionality reduction approaches that have been applied to date to model soil properties with spectroscopy and paves the way for further scientific investigation based on feature selection methods and machine learning.

Discrimination of five commercial Guibourtia wood species using terahertz time domain spectroscopy combined with machine learning approaches

Abstract Terahertz waves hold significant potential for applications in wood identification, owing to their good penetration and distinctive fingerprints in wood. This study focuses on wood samples from five different Guibourtia species as the research objects. The terahertz time-domain spectroscopy (THz-TDS) is employed to acquire the spectroscopic signals of the wood samples and to extract their optical parameter data. The THz refractive indices are dimensionally reduced through principal component analysis (PCA), and three machine learning models, namely partial least squares-discriminant analysis (PLS-DA), random forest (RF), and support vector machine (SVM), are employed to classify the wood of five different Guibourtia species. Time delays of the wood samples from five different Guibourtia species are concentrated in the range of 60–62 ps and exhibit different amplitudes in the frequency domain. Refractive indices showed significant variations within the THz band. PCA for dimensionality reduction of terahertz time-domain spectral data significantly improves the recognition rate of machine learning models. Applying PCA to the refractive index data, the RF model achieves a highest recognition rate of 96.9 % and an overall classification accuracy of 98 %. Current results demonstrate that THz-TDS enables rapid, accurate, and non-destructive classification and identification of wood from the Guibourtia species.

Immediate traceability of ash based on laser-induced breakdown spectroscopy and machine learning

This article reports on an advanced level physical laboratory experiment designed for college level undergraduate education and for scholars who need specialized training on using and interpreting Laser-Induced Breakdown Spectroscopy. The technical principle, experimental operation and sample preparation of Laser-Induced Breakdown Spectroscopy are introduced in detail. A presentation and discussion of the use of Laser-Induced Breakdown Spectroscopy in the traceability of four common samples is emphasized. Combining Laser-Induced Breakdown Spectroscopy with Machine Learning, two distinct datasets are constructed through the extraction of spectral features. Dimensionality reduction of spectral data is performed using Linear Discriminant Analysis, while the Random Forest model is employed for provenance classification. Finally, the interpretability of the Random Forest model is leveraged to explore the contributions of different spectral elements to provenance tracing. Results demonstrated the system’s effectiveness in not only accurately identifying ash types but also in elucidating the influential chemical components, offering significant implications for material analysis and environmental monitoring. On an educational standpoint, this paper will allow any reader, in particular, undergraduate and graduate students, to gain a better understanding of the theory and practice of Laser-Induced Breakdown Spectroscopy and machine learning.

Spatiospectral image processing workflow considerations for advanced MR spectroscopy of the brain.

Magnetic resonance spectroscopy (MRS) is one of the few non-invasive imaging modalities capable of making neurochemical and metabolic measurements in vivo. Traditionally, the clinical utility of MRS has been narrow. The most common use has been the "single-voxel spectroscopy" variant to discern the presence of a lactate peak in the spectra in one location in the brain, typically to evaluate for ischemia in neonates. Thus, the reduction of rich spectral data to a binary variable has not classically necessitated much signal processing. However, scanners have become more powerful and MRS sequences more advanced, increasing data complexity and adding 2 to 3 spatial dimensions in addition to the spectral one. The result is a spatially- and spectrally-variant MRS image ripe for image processing innovation. Despite this potential, the logistics for robustly accessing and manipulating MRS data across different scanners, data formats, and software standards remain unclear. Thus, as research into MRS advances, there is a clear need to better characterize its image processing considerations to facilitate innovation from scientists and engineers. Building on established neuroimaging standards, we describe a framework for manipulating these images that generalizes to the voxel, spectral, and metabolite level across space and multiple imaging sites while integrating with LCModel, a widely used quantitative MRS peak-fitting platform. In doing so, we provide examples to demonstrate the advantages of such a workflow in relation to recent publications and with new data. Overall, we hope our characterizations will lower the barrier of entry to MRS processing for neuroimaging researchers.

Simultaneous Content Determination of Mono-, Di-, and Fructo-oligosaccharides in Citrus Fruit Juices Using an FTIR-PLS Method Based on Selected Absorption Bands.

A quantification method was developed to determine the sugar components, either following addition or enzymatic treatment, in citrus fruit juices containing additional fructo-oligosaccharides using midinfrared spectroscopy. For the quantification, we compared the results obtained by applying the simultaneous equation method, which uses very little wavenumber information, and the partial least squares (PLS) regression method, which requires a lot of wavenumber information. In order to prevent overfitting in the PLS method, we concentrated on reducing the amount of spectral data used in the analysis. The corresponding FTIR-PLS method led to an accurate quantification of the sugar contents, even in enzymatically treated orange juices with complicated compositions. The spectral data used for model calibration were significantly reduced by focusing on the absorption and assignment information of the sugar components. The RMSEs of Glc, Fru, Suc, GF2, and GF3 in enzyme-treated orange juice before and after spectral data reduction were 0.50, 0.46, 0.61, 0.74, and 0.61 g/L and 0.51, 0.49, 0.73, 0.86, and 0.61 g/L, respectively. The developed method could be easily implemented for practical applications, using a simple measuring instrument since only absorption information at the limited absorption bands is required.

Research on intelligent clearing of weeds in wheat fields using spectral imaging and machine learning

The intelligent management of weed clearing can improve wheat yield and reduce the use of pesticides. In this paper, our contribution is to propose a method for wheat and weed recognition based on multispectral imaging and Membrane Search Algorithm Support Vector Machine (MSA-SVM), which is the basis for intelligent weed clearing. A spectral data acquisition system is set up in the laboratory. Then, a total of 700 groups of wheat and weed spectral data are collected. The classical SVM based on spectral data is applied to identify weeds to study the classification performance of wheat and weeds in different bands. The Membrane Search Algorithm (MSA) algorithm proposed by us was used for SVM parameters optimization, and the dimension reduction algorithm was adopted to reduce the interference of redundant information. After local linear embedding (LLE) dimensionality reduction of spectral data, MSA-SVM has the highest average recognition accuracy for weeds and wheat, which is 91.00%. The experimental results show that our research can distinguish wheat from weeds, which is of great significance for the development of smart agriculture.

Inferring Drumhead Damping and Tuning from Sound Using Finite Difference Time Domain (FDTD) Models

Percussionists use a multitude of objects and materials, mounted on their instruments, to achieve a satisfying sound texture. This is a tedious process as there are no guidelines suggesting how to manipulate a percussion instrument to adjust its perceptual characteristics in the desired direction. To this end, the article presents a methodology for computationally identifying how to damp and tune a drumhead by adjusting its mass distribution, e.g., by applying malleable paste on its surface. A dataset of 11,114 sounds has been synthesized using a FDTD solution of the wave equation representing the vibration of a membrane, which is being transmuted through the application of paste. These sounds are investigated to derive conclusions concerning their spectral characteristics and data reduction techniques are used to investigate the feasibility of computationally inferring damping parameters for a given sound. Furthermore, these sounds are used to train a Convolutional Neural Network to infer mass distribution from sound. Results show that computational approaches can provide valuable information to percussionists striving to adjust their personal sound. Although this study has been performed with synthesized sounds, the research methodology presents some inspiring ideas for future investigations with prerecorded sounds.

Mapping Abandoned Uranium Mine Features Using Worldview-3 Imagery in Portions of Karnes, Atascosa and Live Oak Counties, Texas

Worldview-3 (WV3) 16-band multispectral data were used to map exposed bedrock and mine waste piles associated with legacy open-pit mining of sandstone-hosted roll-front uranium deposits along the South Texas Coastal Plain. We used the “spectral hourglass” approach to extract spectral endmembers representative of these features from the image. This approach first requires calibrating the imagery to reflectance, then masking for vegetation, followed by spatial and spectral data reduction using a principal component analysis-based procedure that reduces noise and identifies homogeneous targets which are “pure” enough to be considered spectral endmembers. In this case, we used a single WV3 image which covered an ~11.5 km by ~19.5 km area of Karnes, Atascosa and Live Oak Counties, underlain by mined rocks from the Jackson Group and Catahoula Formation. Up to 58 spectral endmembers were identified using a further multi-dimensional class segregation method and were used as inputs for spectral angle mapper (SAM) classification. SAM classification resulted in the identification of at least 117 mine- and mine waste-related features, most of which were previously unknown. Class similarity was further evaluated, and the dominant minerals in each class were identified by comparison to spectral libraries and measured samples of actual Jackson Group uranium host rocks. Redundant classes were eliminated, and SAM was run a second time using a reduced set of 23 endmembers, which were found to map these same features as effectively as using the full 58 set of endmembers, but with significantly reduced noise and spectral outliers. Our classification results were validated by evaluating detailed scale mapping of three known mine sites (Esse-Spoonamore, Wright-McCrady and Garbysch-Thane) with published ground truth information about the vegetation cover, extent of erosion and exposure of waste pile materials and/or geologic information about host lithology and mineralization. Despite successful demonstration of the utility of WV3 data for inventorying mine features, additional landscape features such as bare agricultural fields and oil and gas drill pads were also identified. The elimination of such features will require combining the spectral classification maps presented in this study with high-quality topographic data. Also, the spectral endmembers identified during the course of this study could be useful for larger-scale mapping efforts using additional well-calibrated WV3 images beyond the coverage of our initial study area.

Automated SpectroPhotometric Image REDuction (ASPIRED)

We provide a suite of public open-source spectral data-reduction software to rapidly obtain scientific products from all forms of long-slit-like spectroscopic observations. Automated SpectroPhotometric REDuction (ASPIRED) is a Python-based spectral data-reduction toolkit. It is designed to be a general toolkit with high flexibility for users to refine and optimize their data-reduction routines for the individual characteristics of their instruments. The default configuration is suitable for low-resolution long-slit spectrometers and provides a quick-look quality output. However, for repeatable science-ready reduced spectral data, some moderate one-time effort is necessary to modify the configuration. Fine-tuning and additional (pre)processing may be required to extend the reduction to systems with more complex setups. It is important to emphasize that although only a few parameters need updating, ensuring their correctness and suitability for generalization to the instrument can take time due to factors such as instrument stability. We compare some example spectra reduced with ASPIRED to published data processed with iraf-based and STARLINK-based pipelines, and find no loss in the quality of the final product. The Python-based, iraf-free ASPIRED can significantly ease the effort of an astronomer in constructing their own data-reduction workflow, enabling simpler solutions to data-reduction automation. This availability of near-real-time, science-ready data will allow adaptive observing strategies, particularly important in, but not limited to, time-domain astronomy.

Nondestructive testing and visualization of compound heavy metals in lettuce leaves using fluorescence hyperspectral imaging

This study evaluated the feasibility of nondestructive testing and visualization of compound heavy metals (cadmium and lead) in lettuce leaves using fluorescence hyperspectral imaging. In addition, a method involving wavelet transform and stepwise regression (WT-SR) was proposed to perform dimensionality reduction of fluorescence spectral data. Fluorescent hyperspectral image acquisition and mathematical analysis were carried out on lettuce leaf samples processed with different compound heavy metal concentrations. The entire lettuce leaf sample was selected as a region of interest (ROI). Savitzky-Golay (SG) algorithm, multivariate scatter correction (MSC), standard normalized variable (SNV), first derivative (1st Der) and second derivative (2nd Der) were used to preprocess the ROI fluorescence spectra. Further, the successive projections algorithm (SPA), the competitive adaptive reweighted sampling (CARS), the iteratively retaining informative variables (IRIV) and variable iterative space shrinkage approach (VISSA), and the wavelet transform combined with stepwise regression (WT-SR) were used to reduce the dimension of spectral data. Finally, the multiple linear regression (MLR) algorithm was used to build the compound heavy metal content detection models. The results showed that the MLR models based on the feature data obtained by 1st Der-WT-SR achieved reasonable performance with Rp2 of 0.7905, RMSEP of 0.0269 mg/kg and RPD of 2.477 for Cd content under wavelet fifth layer decomposition, and with Rp2 of 0.8965, RMSEP of 0.0096 mg/kg and RPD of 3.211 for Pb content under wavelet first layer decomposition. The distribution maps of cadmium and lead contents in lettuce leaves were established using the optimal prediction models. The results further confirmed the great potential of fluorescence hyperspectral technology combined with optimization algorithm for the detection of compound heavy metals.

Application of the combination method based on RF and LE in near infrared spectral modeling

The dimensionality of near-infrared (NIR) spectral data is often extremely large. Dimensionality reduction of spectral data can effectively reduce the redundant information and correlation between spectral variables and simplify the model, which is crucial to increasing the model's performance. As a nonlinear feature extraction method, Laplacian Eigenmaps (LE) may preserve the local neighborhood information of the dataset, has high robustness, and is simple to compute. However, when the LE algorithm maps the data from high-dimensional space to low-dimensional space, it is often disturbed by irrelevant information and multicollinearity in the spectral data, which lowers the model's prediction performance. Random Frog (RF) algorithm can eliminate noise and collinearity in the spectrum. Therefore, before using the LE algorithm, we use the RF algorithm to eliminate irrelevant information in the spectrum and reduce the correlation between the spectra variables to increase the efficiency of the LE algorithm. We used the RF + LE algorithm to reduce the dimensionality of two public NIRS datasets (soil datasets and pharmaceutical tablets datasets) and compared it with RF and LE algorithms alone. We utilized Partial Least Squares Regression (PLSR) and Support Vector Regression (SVR) to establish regression models. The experimental findings demonstrate that compared with the RF algorithm and LE algorithm, the RF + LE combination method can reduce the dimension of spectral variables and model complexity, and improve regression models' prediction accuracy and stability. It is an effective dimensionality reduction method for the near-infrared spectrum.

Spectral noise and data reduction using a long short-term memory network for nonlinear ultrasonic modulation-based fatigue crack detection

This paper presents a spectral noise and data reduction technique based on long short-term memory (LSTM) network for nonlinear ultrasonic modulation-based fatigue crack detection. The amplitudes of the nonlinear modulation components created by a micro fatigue crack are often very small and masked by noise. In addition, the collection of large amounts of data is often undesirable owing to the limited power, data storage, and data transmission bandwidth of monitoring systems. To tackle the issues, an LSTM network was applied to ultrasonic signals to reduce the noise level and the amount of data. The proposed technique offers the following benefits: (1) spectral noise reduction using the LSTM network for ultrasonic signals and (2) data reduction without compromising the spectral density amplitude of the existing nonlinear modulation components. Finally, the performance evaluation was conducted using the data obtained from complex geometry and real structure under external noises, indicating that the proposed method can be applied to various structures.

Analysis of Egg Variation and Foreign Egg Rejection in Rüppell’s Weaver (Ploceus galbula)

Egg appearance is notable for its variation and as a source of recognition cues in bird species that are subject to egg-mimicking brood parasitism. Here I analyze the egg appearance of an East African weaverbird species that has variable eggs and is a host of brood parasitism by an egg-mimicking cuckoo, in order to (1) compare population variation to variation within a clutch as a measure of the distinctiveness of eggs; (2) assess modularity versus correlation among egg appearance traits as an indication of the complexity of egg signatures; and (3) address whether the eggs are discretely polymorphic or continuously variable in appearance. I also compare three methods of assessing egg coloration: reduction of spectral data to orthogonal components, targeted spectral shape variables, and avian visual modeling. Then I report the results of egg replacement experiments that assess the relationship between egg rejection behavior and the difference in appearance between own and foreign eggs. Rüppell’s weaver (Ploceus galbula) eggs are variable in appearance between individuals and consistent within a clutch, but vary widely in the distinctiveness of particular traits. Most aspects of color and spotting are decoupled from each other, including coloration likely to derive from different pigments. Egg ground color is bimodal, with a broad continuous class of off-white/UV eggs and another broad class of blue-green eggs. Variation in all other traits is unimodal and usually normal in distribution. Females reject foreign eggs on the basis of the difference in brightness of the ground color and spotting of foreign eggs relative to their own, and the difference in degree to which spots are aggregated at the broad end of the egg. This aggregation is among the most distinctive features of their eggs, but the brightness of the ground color and spotting brightness are not; the birds’ use of brightness rather than the more distinctive chromatic variation to recognize eggs might reflect the salience of achromatic contrast in a dim enclosed nest.

Detection of heavy metal lead in lettuce leaves based on fluorescence hyperspectral technology combined with deep learning algorithm

The feasibility analysis of fluorescence hyperspectral imaging technology was studied for the detection of lead content in lettuce leaves. Further, Monte Carlo optimized wavelet transform stacked auto-encoders (WT-MC-SAE) was proposed for dimensionality reduction and depth feature extraction of fluorescence spectral data. The fluorescence hyperspectral images of 2800 lettuce leaf samples were selected and the whole lettuce leaf was used as the region of interest (ROI) to extract the fluorescence spectrum. Five different pre-processing algorithms were used to pre-process the original ROI spectral data including standard normalized variable (SNV), first derivative (1st Der), second derivative (2ndDer), third derivative (3rd Der) and fourth derivative (4th Der). Moreover, wavelet transform stacked auto-encoders (WT-SAE) and WT-MC-SAE were used for data dimensionality reduction, and support vector machine regression (SVR) was used for modeling analysis. Among them, 4th Der tends to be the most useful fluorescence spectral data for Pb content detection at 0.067 ∼ 1.400 mg/kg in lettuce leaves, with Rc2 of 0.9802, RMSEC of 0.02321 mg/kg, Rp2 of 0.9467, RMSEP of 0.04017 mg/kg and RPD of 3.273, and model scale (the number of nodes in the input layer, hidden layer and output layer) was 407-314-286-121-76 under the fifth level of wavelet decomposition. Further studies showed that WT-MC-SAE realizes the depth feature extraction of the fluorescence spectrum, and it is of great significance to use fluorescence hyperspectral imaging to realize the quantitative detection of lead in lettuce leaves.

Nitrogen content diagnosis of apple trees canopies using hyperspectral reflectance combined with PLS variable extraction and extreme learning machine

Nitrogen (N) is an important mineral element in apple production. Rapid estimation of apple tree N status is helpful for achieving precise N management. The objective of this work was to explore partial least squares (PLS) regression in dimensional reduction of spectral data and build the diagnostic model. The spectral reflectance data were collected from Fuji apple trees with 4 levels of N fertilizer treatment in the Loess Plateau in 2018 and 2019 using an ASD portable spectroradiometer, and leaf total N content was obtained at the same time. The raw spectra were pretreated using Savitzky-Golay (SG) smoothing and a combination of SG and first-order derivative (SG_FD) or second-order derivative (SG_SD). The samples were divided into a calibration dataset and a prediction dataset using SPXY. Based on 4 factors of PLS regression, including latent variables (LVs), X-loading, variable importance in projection (VIP) and regression coefficients (RC), the 6 methods (LVs, X-loading, VIP_01, VIP_02, RC_01 and RC_02) were derived and used for variable extraction, based on which PLS model and ELM model were established. The results indicated that the spectral data processed by SG_FD had the highest signal-to-noise ratio and was selected for subsequent analysis. The amounts of variables extracted by LVs, X-loading, VIP_01, VIP_02, RC_01 and RC_02 were 6, 11, 18, 305, 26 and 88, respectively. The method of extracting variables with an RC threshold based on the minimum RMSEP (RC_02) could effectively avoid the omission of effective information. The RC_02 method was recommended for related research which required accurate wavelength information as a variable. The variable extraction method based on LVs generated an ELM model with a simple structure. The prediction results showed that the ELM model outperformed the PLS model. The PLS(LVs)_ELM model was the best; R2P, RMSEP and RPD were 0.837, 2.393 and 2.220, respectively. Keywords: partial least square, variable extraction method, extreme learning machine, hyperspectral reflectance, apple tree, canopy nitrogen content DOI: 10.25165/j.ijabe.20211403.6157 Citation: Chen S M, Ma L H, Hu T T, Luo L H, He Q, Zhang S W. Nitrogen content diagnosis of apple trees canopies using hyperspectral reflectance combined with PLS variable extraction and extreme learning machine. Int J Agric & Biol Eng, 2021; 14(3): 181–188.

A novel approach to discriminate transgenic soybean seeds based on terahertz spectroscopy

In qualitative and quantitative terahertz(THz)spectroscopic analyses, reduction and feature extraction of original spectral data are important steps. Due to the parameters, sample preparation, and experimental conditions used in THz time-domain spectroscopy (THz-TDS), the sample absorption lines present different degrees of oscillation and contain certain background noise; therefore spectral data dimension reduction is necessary. Since the existing traditional algorithms, such as principal component analysis, cannot extract useful information from signals, an improved spectral feature extraction method is proposed based on geodesic distance nonlinear reduction and partial least squares regression. Three kinds of transmission spectra of transgenic soybeans are obtained in this experiment. To extract the useful information from the spectral data, principal component analysis (PCA), a locally linear embedding (LLE) algorithm and Floyd's improved LLE algorithm (FLLE) are applied. Multiple linear regression analysis (MLR) and partial least squares regression analysis (PLSR) are performed on the reduced dimensional spectral data. The root mean square error (RMSE) of the FLLE-PLSR algorithm is 0.0079, and the determination coefficient(R) is 0.9966, which are obviously better than those of the PCA-MLR, LLE-MLR, FLLE-MLR, PCA-PLSR and LLE-PLSR algorithms. The proposed method can effectively extract characteristic quantities from the THz spectrum data of transgenic soybeans that have broad application value in agricultural security and food supervision.

Nitrogen content diagnosis of apple trees canopies using hyperspectral reflectance combined with PLS variable extraction and extreme learning machine

Nitrogen (N) is an important mineral element in apple production. Rapid estimation of apple tree N status is helpful for achieving precise N management. The objective of this work was to explore partial least squares (PLS) regression in dimensional reduction of spectral data and build the diagnostic model. The spectral reflectance data were collected from Fuji apple trees with 4 levels of N fertilizer treatment in the Loess Plateau in 2018 and 2019 using an ASD portable spectroradiometer, and leaf total N content was obtained at the same time. The raw spectra were pretreated using Savitzky-Golay (SG) smoothing and a combination of SG and first-order derivative (SG_FD) or second-order derivative (SG_SD). The samples were divided into a calibration dataset and a prediction dataset using SPXY. Based on 4 factors of PLS regression, including latent variables (LVs), X-loading, variable importance in projection (VIP) and regression coefficients (RC), the 6 methods (LVs, X-loading, VIP_01, VIP_02, RC_01 and RC_02) were derived and used for variable extraction, based on which PLS model and ELM model were established. The results indicated that the spectral data processed by SG_FD had the highest signal-to-noise ratio and was selected for subsequent analysis. The amounts of variables extracted by LVs, X-loading, VIP_01, VIP_02, RC_01 and RC_02 were 6, 11, 18, 305, 26 and 88, respectively. The method of extracting variables with an RC threshold based on the minimum RMSEP (RC_02) could effectively avoid the omission of effective information. The RC_02 method was recommended for related research which required accurate wavelength information as a variable. The variable extraction method based on LVs generated an ELM model with a simple structure. The prediction results showed that the ELM model outperformed the PLS model. The PLS(LVs)_ELM model was the best; R2P, RMSEP and RPD were 0.837, 2.393 and 2.220, respectively. Keywords: partial least square, variable extraction method, extreme learning machine, hyperspectral reflectance, apple tree, canopy nitrogen content DOI: 10.25165/j.ijabe.20211403.6157 Citation: Chen S M, Ma L H, Hu T T, Luo L H, He Q, Zhang S W. Nitrogen content diagnosis of apple trees canopies using hyperspectral reflectance combined with PLS variable extraction and extreme learning machine. Int J Agric & Biol Eng, 2021; 14(3): 181–188.

Task-Driven Learned Hyperspectral Data Reduction Using End-to-End Supervised Deep Learning.

An important challenge in hyperspectral imaging tasks is to cope with the large number of spectral bins. Common spectral data reduction methods do not take prior knowledge about the task into account. Consequently, sparsely occurring features that may be essential for the imaging task may not be preserved in the data reduction step. Convolutional neural network (CNN) approaches are capable of learning the specific features relevant to the particular imaging task, but applying them directly to the spectral input data is constrained by the computational efficiency. We propose a novel supervised deep learning approach for combining data reduction and image analysis in an end-to-end architecture. In our approach, the neural network component that performs the reduction is trained such that image features most relevant for the task are preserved in the reduction step. Results for two convolutional neural network architectures and two types of generated datasets show that the proposed Data Reduction CNN (DRCNN) approach can produce more accurate results than existing popular data reduction methods, and can be used in a wide range of problem settings. The integration of knowledge about the task allows for more image compression and higher accuracies compared to standard data reduction methods.

Near-infrared fault detection based on stacked regularized auto-encoder network

Near-infrared spectroscopy as a fast, efficient, non-destructive, and non-polluting technology is employed in this paper to acquire process data at the molecular level to detect the process fault. Due to the high intrinsic dimension, information redundancy, multiple linear correlations between variables and high-sensitiveness of spectral data, stacked auto-encoder as an unsupervised learning method is utilized. In addition, for avoiding over fitting of these auto-encoders and making the model more general, L2 regularization term is employed. To realize the detection and classification of multi-class faults, the softmax regression is adopted as a multi-classification algorithm. Finally, the eligibility of the mentioned neural network is investigated through a crude oil desalination and dehydration process. Nonlinear dimensional reduction of spectral data with minimizing the information loss during data compression, high accuracy, and multi-class fault detection with good visualization of its decision edge are three main advantages of the proposed fault detection method comparing with existing methods.