Fourier Features Research Articles

Nanoparticle Classification Using Frequency Domain Analysis on Resource-Limited Platforms.

A mobile system that can detect viruses in real time is urgently needed, due to the combination of virus emergence and evolution with increasing global travel and transport. A biosensor called PAMONO (for Plasmon Assisted Microscopy of Nano-sized Objects) represents a viable technology for mobile real-time detection of viruses and virus-like particles. It could be used for fast and reliable diagnoses in hospitals, airports, the open air, or other settings. For analysis of the images provided by the sensor, state-of-the-art methods based on convolutional neural networks (CNNs) can achieve high accuracy. However, such computationally intensive methods may not be suitable on most mobile systems. In this work, we propose nanoparticle classification approaches based on frequency domain analysis, which are less resource-intensive. We observe that on average the classification takes 29 s per image for the Fourier features and 17 s for the Haar wavelet features. Although the CNN-based method scores 1–2.5 percentage points higher in classification accuracy, it takes 3370 s per image on the same platform. With these results, we identify and explore the trade-off between resource efficiency and classification performance for nanoparticle classification of images provided by the PAMONO sensor.

Probability Density Rank-Based Quantization for Convex Universal Learning Machines.

The distributions of input data are very important for learning machines, such as the convex universal learning machines (CULMs). The CULMs are a family of universal learning machines with convex optimization. However, the computational complexity is a crucial problem in CULMs, because the dimension of the nonlinear mapping layer (the hidden layer) of the CULMs is usually rather large in complex system modeling. In this article, we propose an efficient quantization method called Probability density Rank-based Quantization (PRQ) to decrease the computational complexity of CULMs. The PRQ ranks the data according to the estimated probability densities and then selects a subset whose elements are equally spaced in the ranked data sequence. We apply the PRQ to kernel ridge regression (KRR) and random Fourier feature recursive least squares (RFF-RLS), which are two typical algorithms of CULMs. The proposed method not only keeps the similarity of data distribution between the code book and data set but also reduces the computational cost by using the kd-tree. Meanwhile, for a given data set, the method yields deterministic quantization results, and it can also exclude the outliers and avoid too many borders in the code book. This brings great convenience to practical applications of the CULMs. The proposed PRQ is evaluated on several real-world benchmark data sets. Experimental results show satisfactory performance of PRQ compared with some state-of-the-art methods.

A Double-Variational Bayesian Framework in Random Fourier Features for Indefinite Kernels.

Random Fourier features (RFFs) have been successfully employed to kernel approximation in large-scale situations. The rationale behind RFF relies on Bochner's theorem, but the condition is too strict and excludes many widely used kernels, e.g., dot-product kernels (violates the shift-invariant condition) and indefinite kernels [violates the positive definite (PD) condition]. In this article, we present a unified RFF framework for indefinite kernel approximation in the reproducing kernel Kreĭn spaces (RKKSs). Besides, our model is also suited to approximate a dot-product kernel on the unit sphere, as it can be transformed into a shift-invariant but indefinite kernel. By the Kolmogorov decomposition scheme, an indefinite kernel in RKKS can be decomposed into the difference of two unknown PD kernels. The spectral distribution of each underlying PD kernel can be formulated as a nonparametric Bayesian Gaussian mixtures model. Based on this, we propose a double-infinite Gaussian mixture model in RFF by placing the Dirichlet process prior. It takes full advantage of high flexibility on the number of components and has the capability of approximating indefinite kernels on a wide scale. In model inference, we develop a non-conjugate variational algorithm with a sub-sampling scheme for the posterior inference. It allows for the non-conjugate case in our model and is quite efficient due to the sub-sampling strategy. Experimental results on several large classification data sets demonstrate the effectiveness of our nonparametric Bayesian model for indefinite kernel approximation when compared to other representative random feature-based methods.

Efficient preference clustering via random fourier features

Approximations based on random Fourier features have recently emerged as an efficient and elegant method for designing large-scale machine learning tasks. Unlike approaches using the Nystrom method, which randomly samples the training examples, we make use of random Fourier features, whose basis functions (i.e., cosine and sine ) are sampled from a distribution independent from the training sample set, to cluster preference data which appears extensively in recommender systems. Firstly, we propose a two-stage preference clustering framework. In this framework, we make use of random Fourier features to map the preference matrix into the feature matrix, soon afterwards, utilize the traditional k-means approach to cluster preference data in the transformed feature space. Compared with traditional preference clustering, our method solves the problem of insufficient memory and greatly improves the efficiency of the operation. Experiments on movie data sets containing 100 000 ratings, show that the proposed method is more effective in clustering accuracy than the Nystrom and k-means, while also achieving better performance than these clustering approaches.

Spatial analysis made easy with linear regression and kernels

Kernel methods are a popular technique for extending linear models to handle non-linear spatial problems via a mapping to an implicit, high-dimensional feature space. While kernel methods are computationally cheaper than an explicit feature mapping, they are still subject to cubic cost on the number of points. Given only a few thousand locations, this computational cost rapidly outstrips the currently available computational power. This paper aims to provide an overview of kernel methods from first-principals (with a focus on ridge regression) and progress to a review of random Fourier features (RFF), a method that enables the scaling of kernel methods to big datasets. We show how the RFF method is capable of approximating the full kernel matrix, providing a significant computational speed-up for a negligible cost to accuracy and can be incorporated into many existing spatial methods using only a few lines of code. We give an example of the implementation of RFFs on a simulated spatial data set to illustrate these properties. Lastly, we summarise the main issues with RFFs and highlight some of the advanced techniques aimed at alleviating them. At each stage, the associated R code is provided.

Random Feature Map-Based Multiple Kernel Fuzzy Clustering with All Feature Weights

Kernel clustering methods are useful to discover the non-linear structures hidden in data, but they suffer from the difficulty of kernel selection and high computational complexity. In this paper, we propose a novel random feature map-based multiple kernel fuzzy clustering method with all feature weights, in which low-rank randomized features of multiple kernels are generated by random Fourier feature map and Quasi-Monte Carlo feature map, and maximum entropy technique is applied to optimize the weights of all feature attributes. The proposed method is effective to extract important kernel and the important attributes of the kernel so as to achieve good clustering results. What is more, compared with conventional kernel clustering methods, our method is much more time-saving and is available to large data sets. The experiments based on various data sets show the superiority and efficiency of the proposed method.

A fast and accurate explicit kernel map

Kernel functions are powerful techniques that have been used successfully in many machine learning algorithms. Explicit kernel maps have emerged as an alternative to standard kernel functions in order to overcome the latter’s scalability issues. An explicit kernel map such as Random Fourier Features (RFF) is a popular method for approximating shift invariant kernels. However, it requires large run time in order to achieve good accuracy. Faster and more accurate variants of it have also been proposed recently. All these methods are still approximations to a shift invariant kernel. Instead of an approximation, we propose a fast, exact and explicit kernel map called Explicit Cosine Map (ECM). The advantage of this exact map is manifested in the form of performance improvements in kernel based algorithms. Furthermore, its explicit nature enables it to be used in streaming applications. Another explicit kernel map called Euler kernel map is also proposed. The effectiveness of both kernel maps is evaluated in the application of streaming Anomaly Detection (AD). The AD results indicate that ECM based algorithm achieves better AD accuracy than previous algorithms, while being faster.

Automatic plankton quantification using deep features

Abstract The study of marine plankton data is vital to monitor the health of the world’s oceans. In recent decades, automatic plankton recognition systems have proved useful to address the vast amount of data collected by specially engineered in situ digital imaging systems. At the beginning, these systems were developed and put into operation using traditional automatic classification techniques, which were fed with hand-designed local image descriptors (such as Fourier features), obtaining quite successful results. In the past few years, there have been many advances in the computer vision community with the rebirth of neural networks. In this paper, we leverage how descriptors computed using convolutional neural networks trained with out-of-domain data are useful to replace hand-designed descriptors in the task of estimating the prevalence of each plankton class in a water sample. To achieve this goal, we have designed a broad set of experiments that show how effective these deep features are when working in combination with state-of-the-art quantification algorithms.

Learning Adaptive Random Features

Random Fourier features are a powerful framework to approximate shift invariant kernels with Monte Carlo integration, which has drawn considerable interest in scaling up kernel-based learning, dimensionality reduction, and information retrieval. In the literature, many sampling schemes have been proposed to improve the approximation performance. However, an interesting theoretic and algorithmic challenge still remains, i.e., how to optimize the design of random Fourier features to achieve good kernel approximation on any input data using a low spectral sampling rate? In this paper, we propose to compute more adaptive random Fourier features with optimized spectral samples (wj’s) and feature weights (pj’s). The learning scheme not only significantly reduces the spectral sampling rate needed for accurate kernel approximation, but also allows joint optimization with any supervised learning framework. We establish generalization bounds using Rademacher complexity, and demonstrate advantages over previous methods. Moreover, our experiments show that the empirical kernel approximation provides effective regularization for supervised learning.

Speech emotion recognition using emotion perception spectral feature

SummarySpeech emotion recognition is an important technique for human‐computer interface applications. Due to contain rich information of emotion, the spectral feature is widely used for emotion recognition. However, the recognition performance is limited because of imprecise extracted rule and uncertain size of resolution of spectral feature. To address this issue, motivated by speech coding, we introduced psychoacoustics model, provided a perception spectral subband partition method for obtaining more precise frequency resolution. Moreover, we also provided a new spectral feature on the divided subband frequency signals. The proposed feature includes emotional perception entropy, spectral inclination, and spectral flatness. Then, a Support Vector Machine classifier is used to recognize emotion categories. The experiment results show that the proposed spectral feature is superior to the traditional MFCC feature, and also better than the state‐of‐the‐art Fourier feature and multi‐resolution amplitude feature.

Cross-domain neural-kernel networks

This paper introduces a novel cross-domain neural-kernel networks architecture for semi-supervised domain adaption problem. The proposed model consists of two stream neural-kernel networks corresponding to the source and target domains which are enriched with a coupling term. Each stream neural-kernel networks follows a combination of neural network layer and an explicit feature map constructed by means of random Fourier features. The introduced coupling term aims at enforcing correlations among the output of the intermediate layers of the two stream networks as well as encouraging the two networks to learn shared representation of the data from both source and target domains. Experimental results are given to illustrate the effectiveness of the proposed approaches on synthetic and real-life datasets.

The Online Random Fourier Features Conjugate Gradient Algorithm

Kernel conjugate gradient (KCG) algorithms have been proposed to improve the convergence rate and filtering accuracy of kernel adaptive filters (KAFs). However, sparsification is still required in the KCG algorithms to curb the growth of network structure for online applications. To address this issue, a novel online random Fourier features conjugate gradient (RFFCG) algorithm based on the minimum mean-square error criterion is proposed by approximating the kernel with the random Fourier features. With no requirement of sparsification, the proposed RFFCG significantly reduces the computational and storage complexities in KAFs. Meanwhile, the RFFCG efficiently uses only one error in the loss function to approach the filtering accuracy of the KAF with sparsification based on all the errors in the loss function. Monte Carlo simulations on short-term chaotic time-series prediction validate the superiorities of the proposed RFFCG algorithm.

Linear Multiple Low-Rank Kernel Based Stationary Gaussian Processes Regression for Time Series

Gaussian processes (GP) for machine learning have been studied systematically over the past two decades and they are by now widely used in a number of diverse applications. However, GP kernel design and the associated hyper-parameter optimization are still hard and to a large extend open problems. In this paper, we consider the task of GP regression for time series modeling and analysis. The underlying stationary kernel can be approximated arbitrarily close by a new proposed grid spectral mixture (GSM) kernel, which turns out to be a linear combination of low-rank sub-kernels. In the case where a large number of the sub-kernels are used, either the Nystr\"{o}m or the random Fourier feature approximations can be adopted to deal efficiently with the computational demands. The unknown GP hyper-parameters consist of the non-negative weights of all sub-kernels as well as the noise variance; their estimation is performed via the maximum-likelihood (ML) estimation framework. Two efficient numerical optimization methods for solving the unknown hyper-parameters are derived, including a sequential majorization-minimization (MM) method and a non-linearly constrained alternating direction of multiplier method (ADMM). The MM matches perfectly with the proven low-rank property of the proposed GSM sub-kernels and turns out to be a part of efficiency, stable, and efficient solver, while the ADMM has the potential to generate better local minimum in terms of the test MSE. Experimental results, based on various classic time series data sets, corroborate that the proposed GSM kernel-based GP regression model outperforms several salient competitors of similar kind in terms of prediction mean-squared-error and numerical stability.

A Random Features-Based Method for Interpolating Digital Terrain Models with High Efficiency

Airborne light detection and ranging (lidar) is becoming a widely adopted technique for capturing elevation data, which are mainly used for creating digital terrain models (DTMs). However, the large size of lidar datasets poses a serious computational challenge to the promising radial basis function (RBF) interpolation method. In this work, to reduce the huge computational cost and improve the interpolation accuracy, random Fourier features are first introduced to approximate the Gaussian kernel of RBFs in feature space, then a random features-based weighted RBF interpolation method is developed. Based on randomized Fourier features, the nonlinear kernel-based training and evaluation of the RBF method is transformed into simple linear operations in feature space, and with the help of weighted ridge regression, the negative effect of the non-Gaussian distribution of lidar datasets on DTM production is reduced. In other words, the combination of randomized Fourier features and weighted ridge regression improves the efficiency and accuracy of the RBF interpolation method. Experiments on simulated datasets indicate that the proposed method performs better than the classical or random features-based RBF method for dealing with non-Gaussian distributed samples, with the former being slightly less accurate than the iterative RBF method due to the low-dimensional random features. However, the computational cost of the new method is much lower compared with the classical or iterative RBFs. Interpolation of airborne lidar-derived points demonstrates that the new method has a computational cost similar to the inverse distance weighting and triangulated irregular network (TIN) approaches, and is significantly faster than the ordinary kriging (OK) or thin plate spline (TPS) methods. Quantitatively, for interpolation of 644,433 points, the proposed method is approximately 833 and 21 times faster than OK and TPS, respectively. Moreover, the new method avoids the surface discontinuity artifacts presented by the OK, TPS, and TIN methods.

Structured-Pump-Enabled Quantum Pattern Recognition.

We report a new scheme of ghost imaging by using a spatially structured pump in the Fourier domain of spontaneous parametric down-conversion for quantum-correlation-based pattern recognition. We exploit the mathematical feature of Laguerre-Gaussian mode's Fourier transform to describe the pump-modulated formation of a ghost image. Of particular interest is the experimental demonstration of a quantum equivalence of a Vander Lugt filter, based on which the nonlocal spiral phase contrast for vortex mapping and quantum-correlation-based human face recognition are implemented successfully. The photons used for probing a test object, scanning the database, and producing a correlation signal can belong to three different light beams, which suggests some security applications where low-light-level illumination and covert operation are desired.

Incremental multi-view spectral clustering

Multi-view learning has attracted increasing attention in recent years, and the existing multi-view learning methods learn a consensus result by collecting all views. These methods have two obvious limitations. First, it is not scalable; with limited computational resources it would be difficult, if not impossible, to collect and process a large collection of views together. Second, in many applications views of data are available over time; it is infeasible to apply the existing multi-view learning methods to such streaming views. To address the two limitations, in this paper we propose a novel incremental multi-view spectral clustering (IMSC) method. In IMSC, instead of ensembling the collection of all views simultaneously, we integrate them one by one in an incremental way. We first learn an initial model from a small number of views; next when a new view is available, we need only use it to update the model and apply the updated model to learn a consensus result. This method is scalable and applicable to streaming views. To further reduce the time and space complexity, we apply low rank approximation by means of the well-known random Fourier features to construct the base kernels and do low rank SVD decompositions accordingly. The theoretical analysis and experimental results on benchmark data sets show that our incremental multi-view spectral clustering method is significantly faster in efficiency than the existing state-of-the-art non-incremental ones and is comparable or even better in clustering quality.

Image retrieval of wool fabric. Part I: Based on low-level texture features

With huge and ever-growing products in the factory, image retrieval can help the worker retrieve the same, or similar, existing products rapidly and accurately to guide production. In this paper, an effective method based on Fourier transform and local binary pattern is proposed to improve the retrieval efficiency of wool fabric. After capturing the fabric image, histogram equalization was implemented on the value of the Hue, Saturation, Value (HSV) mode to enhance the contrast. Subsequently, Fourier transform together with local binary pattern operator were performed to obtain the frequency spectrum and the local binary pattern, respectively. Each frequency spectrum was divided into 22 rings with the same width, and the standard deviation of the frequencies in each ring was calculated as a Fourier feature. Distinct output values of each local binary pattern were counted and normalized as local binary pattern features. Finally, Euclidean distance was adopted to measure the similarity based on the Fourier feature and local binary pattern feature. Twenty thousand wool fabric images were captured to demonstrate the efficacy of the proposed method. Experimental results indicate that the framework is effective and superior for image retrieval of wool fabric, providing referential assistance for the worker in the factory and improving retrieval efficiency.

Shallow neural network with kernel approximation for prediction problems in highly demanding data networks

Intrusion detection and network traffic classification are two of the main research applications of machine learning to highly demanding data networks e.g. IoT/sensors networks. These applications present new prediction challenges and strict requirements to the models applied for prediction. The models must be fast, accurate, flexible and capable of managing large datasets. They must be fast at the training, but mainly at the prediction phase, since inevitable environment changes require constant periodic training, and real-time prediction is mandatory. The models need to be accurate due to the consequences of prediction errors. They need also to be flexible and able to detect complex behaviors, usually encountered in non-linear models and, finally, training and prediction datasets are usually large due to traffic volumes. These requirements present conflicting solutions, between fast and simple shallow linear models and the slower and richer non-linear and deep learning models. Therefore, the perfect solution would be a mixture of both worlds. In this paper, we present such a solution made of a shallow neural network with linear activations plus a feature transformation based on kernel approximation algorithms which provide the necessary richness and non-linear behavior to the whole model. We have studied several kernel approximation algorithms: Nystrom, Random Fourier Features and Fastfood transformation and have applied them to three datasets related to intrusion detection and network traffic classification.This work presents the first application of a shallow linear model plus a kernel approximation to prediction problems with highly demanding network requirements. We show that the prediction performance obtained by these algorithms is positioned in the same range as the best non-linear classifiers, with a significant reduction in computational times, making them appropriate for new highly demanding networks.

Variational Fourier features for Gaussian processes

This work brings together two powerful concepts in Gaussian processes: the variational approach to sparse approximation and the spectral representation of Gaussian processes. This gives rise to an ...

Scalable and efficient learning from crowds with Gaussian processes

Over the last few years, multiply-annotated data has become a very popular source of information. Online platforms such as Amazon Mechanical Turk have revolutionized the labelling process needed for any classification task, sharing the effort between a number of annotators (instead of the classical single expert). This crowdsourcing approach has introduced new challenging problems, such as handling disagreements on the annotated samples or combining the unknown expertise of the annotators. Probabilistic methods, such as Gaussian Processes (GP), have proven successful to model this new crowdsourcing scenario. However, GPs do not scale up well with the training set size, which makes them prohibitive for medium-to-large datasets (beyond 10K training instances). This constitutes a serious limitation for current real-world applications. In this work, we introduce two scalable and efficient GP-based crowdsourcing methods that allow for processing previously-prohibitive datasets. The first one is an efficient and fast approximation to GP with squared exponential (SE) kernel. The second allows for learning a more flexible kernel at the expense of a heavier training (but still scalable to large datasets). Since the latter is not a GP-SE approximation, it can be also considered as a whole new scalable and efficient crowdsourcing method, useful for any dataset size. Both methods use Fourier features and variational inference, can predict the class of new samples, and estimate the expertise of the involved annotators. A complete experimentation compares them with state-of-the-art probabilistic approaches in synthetic and real crowdsourcing datasets of different sizes. They stand out as the best performing approach for large scale problems. Moreover, the second method is competitive with the current state-of-the-art for small datasets.