Spectral Preprocessing Research Articles

A Deep Learning Approach to on-Node Sensor Data Analytics for Mobile or Wearable Devices.

The increasing popularity of wearable devices in recent years means that a diverse range of physiological and functional data can now be captured continuously for applications in sports, wellbeing, and healthcare. This wealth of information requires efficient methods of classification and analysis where deep learning is a promising technique for large-scale data analytics. While deep learning has been successful in implementations that utilize high-performance computing platforms, its use on low-power wearable devices is limited by resource constraints. In this paper, we propose a deep learning methodology, which combines features learned from inertial sensor data together with complementary information from a set of shallow features to enable accurate and real-time activity classification. The design of this combined method aims to overcome some of the limitations present in a typical deep learning framework where on-node computation is required. To optimize the proposed method for real-time on-node computation, spectral domain preprocessing is used before the data are passed onto the deep learning framework. The classification accuracy of our proposed deep learning approach is evaluated against state-of-the-art methods using both laboratory and real world activity datasets. Our results show the validity of the approach on different human activity datasets, outperforming other methods, including the two methods used within our combined pipeline. We also demonstrate that the computation times for the proposed method are consistent with the constraints of real-time on-node processing on smartphones and a wearable sensor platform.

Rapid determination of total protein and wet gluten in commercial wheat flour using siSVR-NIR.

The determination of total protein and wet gluten is of critical importance when screening commercial flour for desired processing suitability. To this end, a near-infrared spectroscopy (NIR) method with support vector regression was developed in the present study. The effects of spectral preprocessing and the synergy interval on model performance were investigated. The results showed that the models from raw spectra were not acceptable, but they were substantially improved by properly applying spectral preprocessing methods. Meanwhile, the synergy interval was validated with a good ability to improve the performance of models based on the whole spectrum. The coefficient of determination (R2), the root mean square error of prediction (RMSEP) and the standard deviation ratio (SDR) of the best models for total protein (wet gluten) were 0.906 (0.850), 0.425 (1.024) and 3.065 (2.482), respectively. These two best models have similar and lower relative errors (approximately 8.8%), which indicates their feasibility.

Automatic Recognition Based on Spectral Preprocessing in Satellite Communication

Based on the spectral characteristic, automatic modulation recognition of satellite communication signal was studied in this paper. Traditional spectral analysis was based on the assumption that mean and variance of background noise spectrum did not vary with frequency. However, in actual satellite communication environment, noise spectrum did not meet the hypothesis. This paper used signal preprocessing by sliding widow least square to weaken the influence of the noise. By analyzing the power spectrum, square spectrum and the forth spectrum of common used modulated signals, the offset ratio was extracted to detect the value of single frequency and a set of characteristic parameters were extracted to recognize signals in the satellite communication without any prior knowledge. Simulation result showed that the method had high practicability, owing to being more resolvable and steady which was not sensitive to the roll-off factor and modulation index of the digital signals. The improved method could also achieve recognition accuracy in the adaptive channel and had certain practical significance.

A Method for Detecting Half-Broken Rotor Bar in Lightly Loaded Induction Motors Using Current

This paper presents an effective method of motor current signature analysis for detecting half- as well as full broken single rotor bar fault of a squirrel-cage induction machine under various loading conditions and speeds. The proposed method is based on spectral preprocessing of the stator current followed by subspace decomposition of the signal autocorrelation matrix to detect relatively low-amplitude fault sidebands. This method is found to be very effective in detecting low-amplitude sinusoids in a signal dominated by high-amplitude fundamental. The extended Kalman filter is used to estimate and track the fundamental component of the stator current. This component is subtracted from the measured stator current at every time step generating a resultant signal with a very low or negligible fundamental component. Subsequently, multiple-signal classification (MUSIC) is applied on the resultant stator current signal. Motor slip is estimated from principle slot harmonic to decide the approximate location of the fault sidebands. For effective fault detection, a hypothesis test is proposed to check the presence of sufficient fault frequency sideband in the current spectrum. This test works better if the lobe in the MUSIC plot due to the fault frequency is not distorted or overlapped by the fundamental component. Therefore, for each data window, the minimum size of the autocorrelation matrix is determined to generate distinct peaks. The proposed method applies to steady-state condition and is found to exhibit superior performance even during the light-load conditions with a half-broken bar.

A Fast Spatial–Spectral Preprocessing Module for Hyperspectral Endmember Extraction

Mixed-pixel decomposition of a hyperspectral image is developed on the basis of extracting unique constituent elements known as endmembers (EMs) and their abundance fraction estimation. Recently, integration of spatial content and spectral information is applied by means of several preprocessing modules (PPs) with the purpose of improving EM extraction (EE) accuracy and decreasing EE time. In this letter, a fast spatial–spectral preprocessing module is proposed, which determines the spectral purity score of pixels located at spatially homogenous regions. These homogenous regions including not spatial border pixels are identified using unsupervised k-means clustering technique and spatial neighborhood system. Afterward, a fraction of homogenous pixels (usually half) with greater spectral purity score is adopted as the best EM candidates for subsequent EEs. This novel PP is examined on synthetic and real AVIRIS data sets, which demonstrates its worthy performance in terms of accuracy and fast computation time.

Measurement of Available Phosphorus and Potassium Contents in Soil using Visible-near-infrared Spectroscopy in Conjunction with SPA-LS-SVM Methods

Applying Near Infrared Reflectance Spectroscopy (NIRS) on farmlands can effectively estimate the available phosphorus and potassium contents of soil online. Spectral preprocessing, including Savitzky Golay (SG), Standard Normal Variate (SNV), Multiplicative Scatter Correction (MSC) and SG 1st derivative, aimed to eliminate system noise and external interference. A correction model was created using respectively Radial Basis Function (RBF) and Least Squares Support Vector Machine (LS-SVM) methods with input from the characteristic wavelengths obtained using Successive Projections Algorithm (SPA). The results of predicting available phosphorus and potassium contents in soil using these two modeling methods were evaluated and the better model was selected. The results showed that the LS-SVM method with input from the characteristic wavelengths obtained using SPA had an advantage over the RBF modeling method. In SPA-LS-SVM models, the correlation coefficient and mean square error of prediction for available phosphorus were 0.8625 and 8.67 and those for available potassium were 0.7843 and 13.42, respectively. This indicates that SPA-based visible-near-infrared spectroscopy using LS-SVM for modeling can be used as a method to accurately measure available phosphorus and potassium contents in soil.

Identification of moisture content in tobacco plant leaves using outlier sample eliminating algorithms and hyperspectral data

Fast identification of moisture content in tobacco plant leaves plays a key role in the tobacco cultivation industry and benefits the management of tobacco plant in the farm. In order to identify moisture content of tobacco plant leaves in a fast and nondestructive way, a method involving Mahalanobis distance coupled with Monte Carlo cross validation(MD-MCCV) was proposed to eliminate outlier sample in this study. The hyperspectral data of 200 tobacco plant leaf samples of 20 moisture gradients were obtained using FieldSpc® 3 spectrometer. Savitzky-Golay smoothing(SG), roughness penalty smoothing(RPS), kernel smoothing(KS) and median smoothing(MS) were used to preprocess the raw spectra. In addition, Mahalanobis distance(MD), Monte Carlo cross validation(MCCV) and Mahalanobis distance coupled to Monte Carlo cross validation(MD-MCCV) were applied to select the outlier sample of the raw spectrum and four smoothing preprocessing spectra. Successive projections algorithm (SPA) was used to extract the most influential wavelengths. Multiple Linear Regression (MLR) was applied to build the prediction models based on preprocessed spectra feature in characteristic wavelengths. The results showed that the preferably four prediction model were MD-MCCV-SG (Rp2 = 0.8401 and RMSEP = 0.1355), MD-MCCV-RPS (Rp2 = 0.8030 and RMSEP = 0.1274), MD-MCCV-KS (Rp2 = 0.8117 and RMSEP = 0.1433), MD-MCCV-MS (Rp2 = 0.9132 and RMSEP = 0.1162). MD-MCCV algorithm performed best among MD algorithm, MCCV algorithm and the method without sample pretreatment algorithm in the eliminating outlier sample from 20 different moisture gradients of tobacco plant leaves and MD-MCCV can be used to eliminate outlier sample in the spectral preprocessing.

Evaluation and optimization of the robustness of a multivariate analysis methodology for identification of alloys by laser induced breakdown spectroscopy

In this study, laser induced breakdown spectroscopy (LIBS) with chemometrics was used for classification and identification of alloys, with a particular focus on the issue of the model robustness. A supervised classification model, Soft Independent Modeling of Class Analogy (SIMCA) was calculated with calibration spectra of 13 representative materials. These measurements were reproduced, with the same samples and using the same LIBS instrument, on two different dates (seven and eight months after the calibration measurements): during this period, instrumental variations occurred and the robustness of sample classification was assessed by the prediction error rate. Then, the optimization of SIMCA model parameters, including spectral preprocessing and wavelength selection, was performed using a full factorial experimental design, and a prediction error rate of 0% with a robustness of 100% was achieved for this period extending until eight months after the model calibration. The study was completed two and a half years later by a test of the robustness of the previously optimized model, carried out with an additional series of measurements on test samples with the same LIBS instrument. The predictive ability of the model on spectra acquired more than two years after validation remained good.

Topological data analysis: A promising big data exploration tool in biology, analytical chemistry and physical chemistry

An important feature of experimental science is that data of various kinds is being produced at an unprecedented rate. This is mainly due to the development of new instrumental concepts and experimental methodologies. It is also clear that the nature of acquired data is significantly different. Indeed in every areas of science, data take the form of always bigger tables, where all but a few of the columns (i.e. variables) turn out to be irrelevant to the questions of interest, and further that we do not necessary know which coordinates are the interesting ones. Big data in our lab of biology, analytical chemistry or physical chemistry is a future that might be closer than any of us suppose. It is in this sense that new tools have to be developed in order to explore and valorize such data sets. Topological data analysis (TDA) is one of these. It was developed recently by topologists who discovered that topological concept could be useful for data analysis. The main objective of this paper is to answer the question why topology is well suited for the analysis of big data set in many areas and even more efficient than conventional data analysis methods. Raman analysis of single bacteria should be providing a good opportunity to demonstrate the potential of TDA for the exploration of various spectroscopic data sets considering different experimental conditions (with high noise level, with/without spectral preprocessing, with wavelength shift, with different spectral resolution, with missing data).

Discrimination of natural gas-related bacteria by means of micro-Raman spectroscopy

Methane-oxidizing bacteria (MOB) are a unique group of gram-negative bacteria that are proved to be biological indicator for gas prospecting since they utilize methane as a sole source of carbon and energy. Herein the feasibility of a novel and efficient gas prospecting method using Raman spectroscopy is studied. Confocal Raman spectroscopy is utilized to establish a Raman database of 11 species of methanotrophs and other closely related bacteria with similar morphology that generally coexist in the upper soil of natural gas. After strict and consistent spectral preprocessing, Raman spectra from the whole cell area are analyzed using the combination of principal component analysis (PCA) and Mahalanobis distance (MD) that allow unambiguous classification of the different cell types with an accuracy of 95.91%. The discrimination model based on multivariate analysis is further evaluated by classifying Raman spectra from independently cultivated bacteria, and achieves an overall accuracy of 94.04% on species level. Our approach using Raman spectroscopy in combination with statistical analysis of various gas reservoirs related bacteria provides rapid distinction that can potentially play a vital role in gas exploration.

Monitoring the liquid phase concentration by Raman spectroscopy in a polymorphic system

In this work, Raman spectroscopy was successfully used for the quantitative determination of the liquid phase concentration in an aqueous polymorphic system of D‐mannitol. An extensive study has initially been performed to identify the influence of the solid state, e.g. particle size, particle amount, and different polymorphs, on the intensity of the characteristic Raman solute signal. It was found that the existence of solid phase can decrease Raman intensity, and this influence is more significant when the suspension density is higher, e.g. with smaller size and larger amount of particles. Based on this information, a large number of samples were examined by Raman spectroscopy in the form of clear solutions and suspensions. The spectral preprocessing and partial least squares (PLS) regression were then used to relate the solute concentrations to these spectral data, independent of solid state. Several PLS calibration models were developed with different treatments to the spectral data, and the optimized strategy was finally demonstrated. Particularly, a reference peak at 578 cm−1 related to the sapphire in the Raman probe window was innovatively applied to reduce the influences from the equipment and other external variations, with which the full‐spectrum PLS model was seen to give more stable results rather than partial spectral regions. The optimized model was subsequently applied to predict the liquid phase concentration in a multiphase multicomponent dynamic process, the solvent mediated polymorphic transformation (SMPT) of mannitol, and it was shown that the offline measurements and the predicted values were mainly in agreement with one another. Copyright © 2015 John Wiley & Sons, Ltd.

Recent Applications of Spectroscopic and Hyperspectral Imaging Techniques with Chemometric Analysis for Rapid Inspection of Microbial Spoilage in Muscle Foods

AbstractMuscle food is one of the most perishable food products because of its vulnerability to microbial spoilage, which can result in critical food safety problems. Traditional techniques for detection and evaluation of microbial spoilage in muscle foods are tedious, laborious, destructive, and time‐consuming. In recent years, spectroscopic and imaging technologies have shown great potentials for the assessment of food quality and safety due to their nondestructive, noninvasive, cost‐effective, and rapid responsive nature. This review focuses on the applications of several valuable spectroscopic techniques including visible and near‐infrared spectroscopy, Fourier transform infrared spectroscopy, fluorescence spectroscopy, Raman spectroscopy, and hyperspectral imaging for the rapid and nondestructive detection of microbial spoilage in common muscle foods such as meat, poultry, fish, and related products. Combined with chemometric analysis, such as spectral preprocessing and modeling methods, these potential technologies have been successfully developed for the determination of total viable count, aerobic plate count, Enterobacteriaceae, Pseudomonas, Escherichia coli, and lactic acid bacteria loads in muscle foods. Moreover, the advantages and disadvantages of these techniques are discussed and some perspectives about future trends are also presented.

Development and integration of block operations for data invariant automation of digital preprocessing and analysis of biological and biomedical Raman spectra.

High-throughput information extraction from large numbers of Raman spectra is becoming an increasingly taxing problem due to the proliferation of new applications enabled using advances in instrumentation. Fortunately, in many of these applications, the entire process can be automated, yielding reproducibly good results with significant time and cost savings. Information extraction consists of two stages, preprocessing and analysis. We focus here on the preprocessing stage, which typically involves several steps, such as calibration, background subtraction, baseline flattening, artifact removal, smoothing, and so on, before the resulting spectra can be further analyzed. Because the results of some of these steps can affect the performance of subsequent ones, attention must be given to the sequencing of steps, the compatibility of these sequences, and the propensity of each step to generate spectral distortions. We outline here important considerations to effect full automation of Raman spectral preprocessing: what is considered full automation; putative general principles to effect full automation; the proper sequencing of processing and analysis steps; conflicts and circularities arising from sequencing; and the need for, and approaches to, preprocessing quality control. These considerations are discussed and illustrated with biological and biomedical examples reflecting both successful and faulty preprocessing.

A rapid qualitative and quantitative evaluation of grape berries at various stages of development using Fourier-transform infrared spectroscopy and multivariate data analysis

Fourier transform (FT) near-infrared (NIR) and attenuated total reflection (ATR) FT mid-infrared (MIR) spectroscopy were used to qualitatively and quantitatively analyse Vitis vinifera L. cv Sauvignon blanc grape berries. FT-NIR and ATR FT-MIR spectroscopy, coupled with spectral preprocessing and multivariate data analysis (MVDA), provided reliable methods to qualitatively assess berry samples at five distinct developmental stages: green, pre-véraison, véraison, post-véraison and ripe (harvest), without any prior metabolite extraction. Compared to NIR spectra, MIR spectra provided more reliable discrimination between the berry samples from the different developmental stages. Interestingly, ATR FT-MIR spectra from fresh homogenized berry samples proved more discriminatory than spectra from frozen homogenized berry samples. Different developmental stages were discriminated by principal component analysis (PCA) and orthogonal partial least squares discriminant analysis (OPLS-DA). In order to generate partial least squares (PLS) models from the MIR/NIR spectral datasets; the major sugars (glucose and fructose) and organic acids (malic acid, succinic acid and tartaric acid) were separated and quantified by high performance liquid chromatography (HPLC) and the data used as a reference dataset. PLS regression was used to develop calibration models to predict the concentration of the major sugars and organic acids in the berry samples from different developmental stages. Our data show that infrared (IR) spectroscopy could provide a rapid, reproducible and cost-effective alternative to the chromatographic analysis of the sugar and organic acid composition of grape berries at various developmental stages, using small sample volumes and requiring limited sample preparation. This provides scope and support for the possible development of hand-held devices to assess quality parameters in field-settings in real-time and non-destructively using IR technologies.

Multi-Target Recognition of Internal and External Defects of Potato by Semi-Transmission Hyperspectral Imaging and Manifold Learning Algorithm

The present paper put forward a non-destructive detection method which combines semi-transmission hyperspectral imaging technology with manifold learning dimension reduction algorithm and least squares support vector machine (LSSVM) to recognize internal and external defects in potatoes simultaneously. Three hundred fifteen potatoes were bought in farmers market as research object, and semi-transmission hyperspectral image acquisition system was constructed to acquire the hyperspectral images of normal external defects (bud and green rind) and internal defect (hollow heart) potatoes. In order to conform to the actual production, defect part is randomly put right, side and back to the acquisition probe when the hyperspectral images of external defects potatoes are acquired. The average spectrums (390-1,040 nm) were extracted from the region of interests for spectral preprocessing. Then three kinds of manifold learning algorithm were respectively utilized to reduce the dimension of spectrum data, including supervised locally linear embedding (SLLE), locally linear embedding (LLE) and isometric mapping (ISOMAP), the low-dimensional data gotten by manifold learning algorithms is used as model input, Error Correcting Output Code (ECOC) and LSSVM were combined to develop the multi-target classification model. By comparing and analyzing results of the three models, we concluded that SLLE is the optimal manifold learning dimension reduction algorithm, and the SLLE-LSSVM model is determined to get the best recognition rate for recognizing internal and external defects potatoes. For test set data, the single recognition rate of normal, bud, green rind and hollow heart potato reached 96.83%, 86.96%, 86.96% and 95% respectively, and he hybrid recognition rate was 93.02%. The results indicate that combining the semi-transmission hyperspectral imaging technology with SLLE-LSSVM is a feasible qualitative analytical method which can simultaneously recognize the internal and external defects potatoes and also provide technical reference for rapid on-line non-destructive detecting of the internal and external defects potatoes.

近红外光谱分析方法研究:从传统数据到大数据

Near-infrared spectroscopy has been widely used in analytical sciences because of its speed, accuracy, environmental friendliness and simplicity. The data type associated with near-infrared spectra is becoming big data because of the development of the process in many fields. The methods used for processing of near-infrared spectra, including spectral preprocessing, outlier detection, multivariate calibration, modeling transfer, and their applications are summarized in this paper. The research approaches for big data analysis of near-infrared spectra, including online detection of industrial products, the identification of products from different batches, and the web of things constructed using near-infrared spectra, are also introduced. Additionally, the prospects for further development of the big data of near-infrared spectra and the cloud platform for near-infrared spectra are discussed.

Cross-scale predictive modeling of CHO cell culture growth and metabolites using Raman spectroscopy and multivariate analysis.

Multi-component, multi-scale Raman spectroscopy modeling results from a monoclonal antibody producing CHO cell culture process including data from two development scales (3 L, 200 L) and a clinical manufacturing scale environment (2,000 L) are presented. Multivariate analysis principles are a critical component to partial least squares (PLS) modeling but can quickly turn into an overly iterative process, thus a simplified protocol is proposed for addressing necessary steps including spectral preprocessing, spectral region selection, and outlier removal to create models exclusively from cell culture process data without the inclusion of spectral data from chemically defined nutrient solutions or targeted component spiking studies. An array of single-scale and combination-scale modeling iterations were generated to evaluate technology capabilities and model scalability. Analysis of prediction errors across models suggests that glucose, lactate, and osmolality are well modeled. Model strength was confirmed via predictive validation and by examining performance similarity across single-scale and combination-scale models. Additionally, accurate predictive models were attained in most cases for viable cell density and total cell density; however, these components exhibited some scale-dependencies that hindered model quality in cross-scale predictions where only development data was used in calibration. Glutamate and ammonium models were also able to achieve accurate predictions in most cases. However, there are differences in the absolute concentration ranges of these components across the datasets of individual bioreactor scales. Thus, glutamate and ammonium PLS models were forced to extrapolate in cases where models were derived from small scale data only but used in cross-scale applications predicting against manufacturing scale batches.

Near-infrared reflectance spectroscopy (NIRS) for rapid determination of ginsenoside Rg1 and Re in Chinese patent medicine Naosaitong pill

Ginsenosides in plant samples have been extensively studied because protopanaxadiol saponins are ubiquitous in Chinese patent medicines, in which they can be used in promoting human health as the main active ingredients. A method for rapid determination of two ginsenosides (Rg1 and Re) in Naosaitong (NST) samples using near-infrared reflectance spectroscopy (NIRS) is studied to determine the contents of ginsenoside Rg1 and Re in this work. Partial least square (PLS) regression was used for building the calibration models, and the effects of spectral preprocessing and variable selection on the models are investigated for optimization of the models. A total of 93 samples were scanned by NIRS, and also by high performance liquid chromatography coupled to a diode array detector to determine the contents of ginsenoside Rg1 and Re. The calibration models for Rg1 and Re had high values of the coefficient of determination (R2) (0.9766 and 0.9764) and low root mean square error of cross validation (RMSECV) (0.0136 and 0.0104), and the values of the standard error of prediction set (SEP) are 0.00764 and 0.0103, which indicate a good correlation between reference values and NIRS predicted values. The overall results show that NIRS could be applied for the rapid determination of the contents of ginsenosides in Ginseng byproducts for pharmaceuticals that develop high-quality Chinese patent medicines.

Soil organic carbon content estimation with laboratory-based visible-near-infrared reflectance spectroscopy: feature selection.

This study, with Yixing (Jiangsu Province, China) and Honghu (Hubei Province, China) as study areas, aimed to compare the successive projection algorithm (SPA) and the genetic algorithm (GA) in spectral feature selection for estimating soil organic carbon (SOC) contents with visible-near-infrared (Vis-NIR) reflectance spectroscopy and further to assess whether the spectral features selected from one site could be applied to another site. The SOC content and Vis-NIR reflectance spectra of soil samples were measured in the laboratory. Savitzky-Golay smoothing and log10(1/R) (R is reflectance) were used for spectral preprocessing. The reflectance spectra were resampled using different spacing intervals ranging from 2 to 10 nm. Then, SPA and GA were conducted for selecting the spectral features of SOC. Partial least square regression (PLSR) with full-spectrum PLSR and the spectral features selected by SPA (SPA-PLSR) and GA (GA-PLSR) were calibrated and validated using independent datasets, respectively. Moreover, the spectral features selected from one study area were applied to another area. Study results showed that, for the two study areas, the SPA-PLSR and GA-PLSR improved estimation accuracies and reduced spectral variables compared with the full spectrum PLSR in estimating SOC contents; GA-PLSR obtained better estimation results than SPA-PLSR, whereas SPA was simpler than GA, and the spectral features selected from Yixing could be well applied to Honghu, but not the reverse. These results indicated that the SPA and GA could reduce the spectral variables and improve the performance of PLSR model and that GA performed better than SPA in estimating SOC contents. However, SPA is simpler and time-saving compared with GA in selecting the spectral features of SOC. The spectral features selected from one dataset could be applied to a target dataset when the dataset contains sufficient information adequately describing the variability of samples of the target dataset.

Analysis of Data Mining Method for Near Infrared Spectral Data of Dairy Products Based Orthogonality

There are advantages such as real time, quick, multi-constituents simultaneous measurement, green and no pollution, and so on, of using near infrared spectral method to detect concentration information of dairy products constituents. From time domain and frequency domain, near infrared spectral data mining methods based on orthogonality were separately researched to realize noise filtering and useful information extracting. For milk spectrum, methods in the time domain including Orthogonal Signal Correction and Direct Orthogonalization, Wavelet Transform de-noising method in the frequency domain, were separately explored and used in spectral preprocessing, PLS method was used to build calibration model, and processing effect of different data mining methods was compared with and analyzed. It is showed that in the time domain, using the orthogonality methods to make data mining for milk spectrum, the complex interfering signal and noise information uncorrelated with the measured constituents can be can reduced effectively, and the correlated information with the measured constituents is reserved in maximal limitation, prediction capability of calibration model is improved, furthermore, the Orthogonal Signal Correction method is better than the Direct Orthogonalization method, moreover, in the frequency domain, wavelet de-noising is better than the orthogonality methods within the time domain.