Nonlinear Feature Space Research Articles

Dual force hybrid detector for hyperspectral images

This letter proposes a novel dual force hybrid detector for hyperspectral image (HSI) target detection. As a hybrid detector, we conduct the HSI spectral unmixing in a non-linear feature space mapped by locally linear embedding, and the image reconstruction error in the manifold embedded feature is considered as the primary force for detection. Then, we introduce the adaptive cosine/coherent estimate detector as the additional force. By combining the aforementioned two parts of detection results together, we obtain the final detection probability map. Based on the public data sets of (1) implanted metal detection in a hyperspectral digital imagery collection experiment image and (2) fabrics and vehicles detection in a Hyperspectral Mapper image, the technical results show that our method achieves superior performance compared to some state-of-the-art hyperspectral detectors.

Quality relevant nonlinear batch process performance monitoring using a kernel based multiway non-Gaussian latent subspace projection approach

Multiway kernel partial least squares method (MKPLS) has recently been developed for monitoring the operational performance of nonlinear batch or semi-batch processes. It has strong capability to handle batch trajectories and nonlinear process dynamics, which cannot be effectively dealt with by traditional multiway partial least squares (MPLS) technique. However, MKPLS method may not be effective in capturing significant non-Gaussian features of batch processes because only the second-order statistics instead of higher-order statistics are taken into account in the underlying model. On the other hand, multiway kernel independent component analysis (MKICA) has been proposed for nonlinear batch process monitoring and fault detection. Different from MKPLS, MKICA can extract not only nonlinear but also non-Gaussian features through maximizing the higher-order statistic of negentropy instead of second-order statistic of covariance within the high-dimensional kernel space. Nevertheless, MKICA based process monitoring approaches may not be well suited in many batch processes because only process measurement variables are utilized while quality variables are not considered in the multivariate models. In this paper, a novel multiway kernel based quality relevant non-Gaussian latent subspace projection (MKQNGLSP) approach is proposed in order to monitor the operational performance of batch processes with nonlinear and non-Gaussian dynamics by combining measurement and quality variables. First, both process measurement and quality variables are projected onto high-dimensional nonlinear kernel feature spaces, respectively. Then, the multidimensional latent directions within kernel feature subspaces corresponding to measurement and quality variables are concurrently searched for so that the maximized mutual information between the measurement and quality spaces is obtained. The I2 and SPE monitoring indices within the extracted latent subspaces are further defined to capture batch process faults resulting in abnormal product quality. The proposed MKQNGLSP method is applied to a fed-batch penicillin fermentation process and the operational performance monitoring results demonstrate the superiority of the developed method as apposed to the MKPLS based process monitoring approach.

Kernel Coupled Cross-Regression for Low-Resolution Face Recognition

Low resolution (LR) in face recognition (FR) surveillance applications will cause the problem of dimensional mismatch between LR image and its high-resolution (HR) template. In this paper, a novel method called kernel coupled cross-regression (KCCR) is proposed to deal with this problem. Instead of processing in the original observing space directly, KCCR projects LR and HR face images into a unified nonlinear embedding feature space using kernel coupled mappings and graph embedding. Spectral regression is further employed to improve the generalization performance and reduce the time complexity. Meanwhile, cross-regression is developed to fully utilize the HR embedding to increase the information of the LR space, thus to improve the recognition performance. Experiments on the FERET and CMU PIE face database show that KCCR outperforms the existing structure-based methods in terms of recognition rate as well as time complexity.

Combination of activation functions in extreme learning machines for multivariate calibration

The key point in multivariate calibration is to build an accurate regression relationship between the predictors and responses. In this paper, we first use extreme learning machine (ELM) to build spectroscopy regression model. Then, we propose a combinational ELM (CELM) method in which the decision function is represented as a sum of a linear hidden-node output function (activation function) and a nonlinear hidden-node output function. As the output functions map the input spectral signal to linear and nonlinear feature spaces respectively, the proposed method can effectively describe the linear and nonlinear relations existed in spectroscopy regression by the CELM output weights vector which can be simply resolved by ridge least squares or alternative iterative regularization. The proposed method is compared, in terms of RMSEP, to PLS and ELM on simulated and real NIR data sets. Experimental results demonstrate the efficacy and effectiveness of the proposed method.

Hyperdisk based large margin classifier

We introduce a large margin linear binary classification framework that approximates each class with a hyperdisk – the intersection of the affine support and the bounding hypersphere of its training samples in feature space – and then finds the linear classifier that maximizes the margin separating the two hyperdisks. We contrast this with Support Vector Machines (SVMs), which find the maximum-margin separator of the pointwise convex hulls of the training samples, arguing that replacing convex hulls with looser convex class models such as hyperdisks provides safer margin estimates that improve the accuracy on some problems. Both the hyperdisks and their separators are found by solving simple quadratic programs. The method is extended to nonlinear feature spaces using the kernel trick, and multi-class problems are dealt with by combining binary classifiers in the same ways as for SVMs. Experiments on a range of data sets show that the method compares favourably with other popular large margin classifiers.

Multiway Gaussian Mixture Model Based Adaptive Kernel Partial Least Squares Regression Method for Soft Sensor Estimation and Reliable Quality Prediction of Nonlinear Multiphase Batch Processes

The predictive model based soft sensor technique has become increasingly important to provide reliable online measurements, facilitate advanced process control and optimization, and improve product quality in process industries. The conventional soft sensors are normally single-model based and thus may not be appropriate for processes with shifting operating phases or conditions and the underlying changing dynamics. In this study, a multiway Gaussian mixture model (MGMM) based adaptive kernel partial least-squares (AKPLS) method is proposed to handle online quality prediction of batch or semibatch processes with multiple operating phases. The three-dimensional measurement data are first preprocessed and unfolded into two-dimensional matrix. Then, the multiway Gaussian mixture model is estimated in order to identify and isolate different operating phases. Further, the process and quality measurements are separated into multiple segments corresponding to those identified phases, and the various localized kernel PLS models are built in the high-dimensional nonlinear feature space to characterize the shifting dynamics across different operating phases. Using Bayesian inference strategy, each process measurement sample of a new batch is classified into a particular phase with the maximal posterior probability, and thus, the local kernel PLS model representing the identical phase can be adaptively chosen for online quality variable prediction. The presented soft sensor modeling method is applied to a simulated multiphase penicillin fermentation process, and the computational results demonstrate that the proposed MGMM-AKPLS approach is superior to the conventional kernel PLS model in terms of prediction accuracy and model reliability.

A learning-based target decomposition method using Kernel KSVD for polarimetric SAR image classification

In this article, a learning-based target decomposition method based on Kernel K-singular vector decomposition (Kernel KSVD) algorithm is proposed for polarimetric synthetic aperture radar (PolSAR) image classification. With new methods offering increased resolution, more details (structures and objects) could be exploited in the SAR images, thus invalidating the traditional decompositions based on specific scattering mechanisms offering low-resolution SAR image classification. Instead of adopting fixed bases corresponding to the known scattering mechanisms, we propose a learning-based decomposition method for generating adaptive bases and developing a nonlinear extension of the KSVD algorithm in a nonlinear feature space, called as Kernel KSVD. It is an iterative method that alternates between sparse coding in the kernel feature space based on the nonlinear dictionary and a process of updating each atom in the dictionary. The Kernel KSVD-based decomposition not only generates a stable and adaptive representation of the images but also establishes a curvilinear coordinate that goes along the flow of nonlinear polarimetric features. This proposed approach was verified on two sets of SAR data and found to outperform traditional decompositions based on scattering mechanisms.

Training the max-margin sequence model with the relaxed slack variables

Sequence models are widely used in many applications such as natural language processing, information extraction and optical character recognition, etc. We propose a new approach to train the max-margin based sequence model by relaxing the slack variables in this paper. With the canonical feature mapping definition, the relaxed problem is solved by training a multiclass Support Vector Machine (SVM). Compared with the state-of-the-art solutions for the sequence learning, the new method has the following advantages: firstly, the sequence training problem is transformed into a multiclassification problem, which is more widely studied and already has quite a few off-the-shelf training packages; secondly, this new approach reduces the complexity of training significantly and achieves comparable prediction performance compared with the existing sequence models; thirdly, when the size of training data is limited, by assigning different slack variables to different microlabel pairs, the new method can use the discriminative information more frugally and produces more reliable model; last but not least, by employing kernels in the intermediate multiclass SVM, nonlinear feature space can be easily explored. Experimental results on the task of named entity recognition, information extraction and handwritten letter recognition with the public datasets illustrate the efficiency and effectiveness of our method.

Human Activity as a Manifold-Valued Random Process

Most of previous shape based human activity models were built with either a linear assumption or an extrinsic interpretation of the nonlinear geometry of the shape space, both of which proved to be problematic on account of the nonlinear intrinsic geometry of the associated shape spaces. In this paper we propose an intrinsic stochastic modeling of human activity on a shape manifold. More importantly, within an elegant and theoretically sound framework, our work effectively bridges the nonlinear modeling of human activity on a nonlinear space, with the classic stochastic modeling in a Euclidean space, and thereby provides a foundation for a more effective and accurate analysis of the nonlinear feature space of activity models. From a video sequence, human activity is extracted as a sequence of shapes. Such a sequence is considered as one realization of a random process on a shape manifold. Different activities are then modeled as manifold valued random processes with different distributions. To address the problem of stochastic modeling on a manifold, we first construct a nonlinear invertible map of a manifold valued process to a Euclidean process. The resulting process is then modeled as a global or piecewise Brownian motion. The mapping from a manifold to a Euclidean space is known as a stochastic development. The advantage of such a technique is that it yields a one-one correspondence, and the resulting Euclidean process intrinsically captures the curvature on the original manifold. The proposed algorithm is validated on two activity databases [15], [5] and compared with the related works on each of these. The substantiating results demonstrate the viability and high accuracy of our modeling technique in characterizing and classifying different activities.

Effect of Physical Contact (Hand-Holding) on Heart Rate Variability

Autonomic nervous system (ANS), Hand- holding, Heart rate variability (HRV), Power spectral density analysis. Abstract Heart's electric field can be measured anywhere on the surface of the body (ECG). When individuals touch, one person's ECG signal can be registered in other person's EEG and elsewhere on his body. Now, the aim of this study was to test the hypothesis that physical contact (hand-holding) of two persons changes their heart rate variability. Subjects were sixteen healthy female (age: 20- 26) which divided into eight sets. In each sets, we had two friends that they passed intimacy test of J.sternberg. ECG of two subjects (each set) acquired for 5 minutes before hand-holding (as control group) and 5 minutes during they held their hands (as experimental group). Then heart rate variability signals were extracted from subjects' ECG and analyzed in linear feature space (time and frequency domain) and nonlinear feature space. Considering the results, we conclude that physical contact (hand-holding of two friends) increases parasympathetic activity, as indicate by increase SD1, SD1/SD2, HF and MF power (p<0.05) and decreases sympathetic activity, as indicate by decrease LF power (p<0.01) and LF/HF ratio (p<0.05).  patterns in our body, and possessing an extensive communication system with the brain than do other major organs, the heart exerts a unique and far-reaching influence on the brain and the entire body (3). The study of communication pathways between brain and heart has been approached from a rather one-sided perspective, with scientists focusing primarily on the heart's responses to the brain's commands. Now, we have learned that communication between the brain and heart is actually a dynamic, ongoing, two-way dialog, with each organ continuously influencing the other's function. The heart communicates with the brain in four main ways: biophysical communication (pulse wave), neurological communication (nervous system), biochemical communication (hormones) and Energetic communication (electromagnetic fields) (4). The brain also sends signal to the heart with two branches of autonomic nervous system (ANS).

Fractality and a Wavelet-Chaos-Neural Network Methodology for EEG-Based Diagnosis of Autistic Spectrum Disorder

A method is presented for investigation of EEG of children with autistic spectrum disorder using complexity and chaos theory with the goal of discovering a nonlinear feature space. Fractal Dimension is proposed for investigation of complexity and dynamical changes in autistic spectrum disorder in brain. Two methods are investigated for computation of fractal dimension: Higuchi's Fractal Dimension and Katz's Fractal Dimension. A wavelet-chaos-neural network methodology is presented for automated EEG-based diagnosis of autistic spectrum disorder. The model is tested on a database of eyes-closed EEG data obtained from two groups: nine autistic spectrum disorder children, 6 to 13 years old, and eight non-autistic spectrum disorder children, 7 to 13 years old. Using a radial basis function classifier, an accuracy of 90% was achieved based on the most significant features discovered via analysis of variation statistical test, which are three Katz's Fractal Dimensions in delta (of loci Fp2 and C3) and gamma (of locus T6) EEG sub-bands with P < 0.001.

Large-Margin Classification in Infinite Neural Networks

We introduce a new family of positive-definite kernels for large margin classification in support vector machines (SVMs). These kernels mimic the computation in large neural networks with one layer of hidden units. We also show how to derive new kernels, by recursive composition, that may be viewed as mapping their inputs through a series of nonlinear feature spaces. These recursively derived kernels mimic the computation in deep networks with multiple hidden layers. We evaluate SVMs with these kernels on problems designed to illustrate the advantages of deep architectures. Compared to previous benchmarks, we find that on some problems, these SVMs yield state-of-the-art results, beating not only other SVMs but also deep belief nets.

Exploring nonlinear feature space dimension reduction and data representation in breast CADx with Laplacian eigenmaps and ‐SNE

In this preliminary study, recently developed unsupervised nonlinear dimension reduction (DR) and data representation techniques were applied to computer-extracted breast lesion feature spaces across three separate imaging modalities: Ultrasound (U.S.) with 1126 cases, dynamic contrast enhanced magnetic resonance imaging with 356 cases, and full-field digital mammography with 245 cases. Two methods for nonlinear DR were explored: Laplacian eigenmaps [M. Belkin and P. Niyogi, "Laplacian eigenmaps for dimensionality reduction and data representation," Neural Comput. 15, 1373-1396 (2003)] and t-distributed stochastic neighbor embedding (t-SNE) [L. van der Maaten and G. Hinton, "Visualizing data using t-SNE," J. Mach. Learn. Res. 9, 2579-2605 (2008)]. These methods attempt to map originally high dimensional feature spaces to more human interpretable lower dimensional spaces while preserving both local and global information. The properties of these methods as applied to breast computer-aided diagnosis (CADx) were evaluated in the context of malignancy classification performance as well as in the visual inspection of the sparseness within the two-dimensional and three-dimensional mappings. Classification performance was estimated by using the reduced dimension mapped feature output as input into both linear and nonlinear classifiers: Markov chain Monte Carlo based Bayesian artificial neural network (MCMC-BANN) and linear discriminant analysis. The new techniques were compared to previously developed breast CADx methodologies, including automatic relevance determination and linear stepwise (LSW) feature selection, as well as a linear DR method based on principal component analysis. Using ROC analysis and 0.632+bootstrap validation, 95% empirical confidence intervals were computed for the each classifier's AUC performance. In the large U.S. data set, sample high performance results include, AUC0.632+ = 0.88 with 95% empirical bootstrap interval [0.787;0.895] for 13 ARD selected features and AUC0.632+ = 0.87 with interval [0.817;0.906] for four LSW selected features compared to 4D t-SNE mapping (from the original 81D feature space) giving AUC0.632+ = 0.90 with interval [0.847;0.919], all using the MCMC-BANN. Preliminary results appear to indicate capability for the new methods to match or exceed classification performance of current advanced breast lesion CADx algorithms. While not appropriate as a complete replacement of feature selection in CADx problems, DR techniques offer a complementary approach, which can aid elucidation of additional properties associated with the data. Specifically, the new techniques were shown to possess the added benefit of delivering sparse lower dimensional representations for visual interpretation, revealing intricate data structure of the feature space.

WE-D-304A-02: Exploring Non-Linear Feature Space Dimension Reduction and Data Representation in Breast CADx

Purpose: Preliminary study to explore potential of recently developed non‐linear dimension reduction and data representation techniques as applied to breast lesion CADx generated feature spaces. Method and Materials: Two new methods were explored: Laplacian eignenmaps of (Belkin and Niyogi) and t‐distributed stochastic neighbor embedding (t‐SNE) (van der Maaten and Hinton). The properties of these methods were evaluated in the context of malignancy classification performance as well as visual inspection of the sparseness for two and three dimensional mapping representations. The robustness of the proposed techniques were tested against four separate imaging modality feature databases, including 2956 ultrasound(US), 735 full‐field digital mammography (FFDM), 850 screen‐film mammography (SFM), and 356 DCE‐MRI biopsy proven breast mass lesions. Using the reduced dimension mapped feature output as input, as opposed to feature selection, two classifiers, linear and non‐linear, were tested: Markov Chain Monte Carlo based Bayesian artificial neural network (MCMC‐BANN) and linear discriminate analysis (LDA). To evaluate performance, the AUC was estimated for each classifier using ROC analysis and 0.632+ bootstrap validation on over 500 samples. In addition, performance was compared to a non‐linear Automatic Relevance Determination (ARD), linear step‐wise feature selection method, as well as a linear reduction Principle Component Analysis based method. Results: The new methods were found to match or exceed performance of current state‐of‐the‐art breast lesion CADx algorithms for feature selection and classification across all modalities. Additionally, the new techniques possess the added benefit of naturally delivering sparse lower dimensional representations for visual interpretation, thus, revealing intricate data structure of the feature space. Conclusion: These methods maintain predictive power while also preserving both local and global structural information present in the original high dimensional feature space for breast lesion feature data used in CADx.Conflict of Interest: M. Giger is a stockholder and receives royalties from Hologic.

Multiway kernel independent component analysis based on feature samples for batch process monitoring

Most batch processes generally exhibit the characteristics of nonlinear variation. In this paper, a nonlinear monitoring technique is proposed using a multiway kernel independent component analysis based on feature samples (FS-MKICA). This approach first unfolds the three-way dataset of a batch process into the two-way one and then chooses representative feature samples from the large two-way input training dataset. The nonlinear feature space abstracted from the unfolded two-way data space is then transformed into a high-dimensional linear space via kernel function and an independent component analysis (ICA) model is established in the mapped linear space. The proposed FS-MKICA method can significantly reduce the computational cost in extracting the kernel ICA model since it is based on the small subset of feature samples rather than on the entire input sample set. We supply two statistics, the I 2 statistic of process variation and the squared prediction error statistic of residual, for on-line monitoring of batch processes. The proposed method is applied to detecting faults in the fed-batch penicillin fermentation process. The standard linear ICA method for batch process monitoring, known as the multiway independent component analysis (MICA), is also applied to the same benchmark batch process. The simulation results obtained in this nonlinear batch process application clearly demonstrate the power and superiority of the new nonlinear monitoring method over the linear one. The FS-MKICA approach can extract the nonlinear features of the batch process while the MICA method cannot.

Support Vector Driven Genetic Algorithm for the Design of Circular Polarized Microstrip Antenna

In this paper, a hybrid soft computing method for designing specific microstrip antenna is presented. Evolutionary algorithm such as genetic algorithm (GA) is one of the promising ways of finding global optimum solution from a multivariate nonlinear feature space. Being a stochastic iterative algorithm, it requires much computation power when the function to be optimized is complex and time consuming. Various meta-modelling techniques such as neural network, response surface methods, kriging, etc. can be used to model the process under optimization in order to reduce the computational expenses. In this paper, we investigate one such technique – support vector regression (SVR) – to model the complex analytical process. The model, thus obtained, is used for optimization using genetic algorithms. This approach is demonstrated for the design of circular polarized microstrip antenna at 2.6 GHz band. The results of SVR model are compared with other meta-models generated with neural network and response surface methodology.

On Feature Extraction via Kernels

Using the kernel trick idea and the kernels-as-features idea, we can construct two kinds of nonlinear feature spaces, where linear feature extraction algorithms can be employed to extract nonlinear features. In this correspondence, we study the relationship between the two kernel ideas applied to certain feature extraction algorithms such as linear discriminant analysis, principal component analysis, and canonical correlation analysis. We provide a rigorous theoretical analysis and show that they are equivalent up to different scalings on each feature. These results provide a better understanding of the kernel method.

Tracking control of nonlinear lumped mechanical continuous-time systems: A model-based iterative learning approach

This paper presents a nonlinear model-based iterative learning control procedure to achieve accurate tracking control for nonlinear lumped mechanical continuous-time systems. The model structure used in this iterative learning control procedure is new and combines a linear state space model and a nonlinear feature space transformation. An intuitive two-step iterative algorithm to identify the model parameters is presented. It alternates between the estimation of the linear and the nonlinear model part. It is assumed that besides the input and output signals also the full state vector of the system is available for identification. A measurement and signal processing procedure to estimate these signals for lumped mechanical systems is presented. The iterative learning control procedure relies on the calculation of the input that generates a given model output, so-called offline model inversion. A new offline nonlinear model inversion method for continuous-time, nonlinear time-invariant, state space models based on Newton's method is presented and applied to the new model structure. This model inversion method is not restricted to minimum phase models. It requires only calculation of the first order derivatives of the state space model and is applicable to multivariable models. For periodic reference signals the method yields a compact implementation in the frequency domain. Moreover it is shown that a bandwidth can be specified up to which learning is allowed when using this inversion method in the iterative learning control procedure. Experimental results for a nonlinear single-input-single-output system corresponding to a quarter car on a hydraulic test rig are presented. It is shown that the new nonlinear approach outperforms the linear iterative learning control approach which is currently used in the automotive industry on durability test rigs.

Modelling of chaotic systems using wavelet kernel partial least squares regression method

Based on the powerful nonlinear mapping ability of kernel learning, and in combination with the partial least square (PLS) algorithm for linear regression, a wavelet kernel partial least square (WKPLS) regression method is proposed. By the method, the input-output data are firstly mapped to a nonlinear higher dimensional feature space, a linear PLS regression model is then constructed by the classic kernel transformation trick used in support vector machines. The PLS approach utilizes the covariance between input and output variables to extract latent features, and the wavelet kernel which is an admissible support vector kernel function is characterized by its local analysis and approximate orthogonality. Hence, the proposed WKPLS method combining PLS approach with wavelet kernel function shows excellent learning performance for modeling nonlinear dynamic systems. The WKPLS is then applied to modelling of several chaotic dynamical systems and compared with the kernel partial least squares(KPLS) method using Gaussian kernel function. Simulation results confirm that the WKPLS identifier is fast and can accurately approximate unknown chaotic dynamical system, and its approximation accuracy is higher than the KPLS under the same conditions.

KERNEL GBDA FOR RELEVANCE FEEDBACK IN IMAGE RETRIEVAL

Relevance feedback, as a user-in-the-loop mechanism, has been widely employed to improve the performance of content-based image retrieval. Generally, in a relevance feedback algorithm, two key components are: (1) how to select a subset of effective features from a large-scale feature pool and, (2) correspondingly, how to construct a suitable dissimilarity measure. In previous work, the biased discriminant analysis (BDA) has been proposed to address these two problems during the feedback iterations. However, BDA encounters the so called small samples size problem because it has a lack of training samples. In this paper, we utilize the generalized singular value decomposition (GSVD) to significantly reduce the small samples size problem in BDA. The developed algorithm is named GSVD for BDA (GBDA). We then kernelize the GBDA to nonlinear kernel feature space. A large amount of experiments were carried out upon a large scale database, which contains 17800 images. From the experimental results, GBDA and its kernelization are demonstrated to outperform the traditional BDA-based relevance feedback approaches and their kernel extensions, respectively.