Fourier Features Research Articles

Major advancements in kernel function approximation

Kernel based methods have become popular in a wide variety of machine learning tasks. They rely on the computation of kernel functions, which implicitly transform the data in its input space to data in a very high dimensional space. Efficient application of these functions have been subject to study in the last 10 years. The main focus was on improving the scalability of kernel based methods. In this regard, kernel function approximation using explicit feature maps have emerged as a substitute for traditional kernel based methods. Over the years, various advancements from the theoretical perspective have been made to explicit kernel maps, especially to the method of random Fourier features (RFF), which is the main focus of our work. In this work, the major developments in the theory of kernel function approximation are reviewed in a systematic manner and the practical applications are discussed. Furthermore, we identify the shortcomings of the current research, and discuss possible avenues for future work.

RFF Based Detection for SCMA in Presence of PA Nonlinearity

The next-generation of communication systems must be capable of serving a significantly higher traffic generated by a large number of devices. To support massive connectivity over limited time-frequency resources, several non-orthogonal multiple access (NOMA) paradigms have emerged. In this context, sparse code multiple access (SCMA) based NOMA is particularly attractive due to its robustness to error-propagation, and its potentially high coding-gain with appropriate codebook design. However, the performance of SCMA based systems is severely degraded by distortions introduced by non-ideal hardware, e.g. power-amplifier (PA) nonlinearity. To mitigate such artefacts, in this letter, a random Fourier feature (RFF) based hybrid message passing algorithm (MPA) is proposed, and validated through computer simulations. Lastly, an analytical proof is presented that indicates that the use of RFFs significantly improves the convergence of the MPA in the presence of impairments, and renders error-rate performance equivalent to that of a linear Gaussian channel under approximate MPA.

Randomized independent component analysis and linear discriminant analysis dimensionality reduction methods for hyperspectral image classification

Dimensionality reduction is an important step in processing the hyperspectral images (HSI) to overcome the curse of dimensionality problem. Linear dimensionality reduction methods such as Independent component analysis (ICA) and Linear discriminant analysis (LDA) are commonly employed to reduce the dimensionality of HSI. These methods fail to capture non-linear dependency in the HSI data, as data lies in the nonlinear manifold. To handle this, nonlinear transformation techniques based on kernel methods were introduced for dimensionality reduction of HSI. However, the kernel methods involve cubic computational complexity while computing the kernel matrix, and thus its potential cannot be explored when the number of pixels (samples) are large. In literature a fewer number of pixels are randomly selected to partial to overcome this issue, however this sub-optimal strategy might neglect important information in the HSI. In this paper, we propose randomized solutions to the ICA and LDA dimensionality reduction methods using Random Fourier features, and we label them as RFFICA and RFFLDA. Our proposed method overcomes the scalability issue and to handle the non-linearities present in the data more efficiently. Experiments conducted with two real-world hyperspectral datasets demonstrates that our proposed randomized methods outperform the conventional kernel ICA and kernel LDA in terms overall, per-class accuracies and computational time.

Multikernel adaptive filtering based on random features approximation

Multikernel adaptive filters (MKAFs) have been proposed to address the selection issue of kernel parameter in single kernel based kernel adaptive filters (KAFs). However, the conventional MKAFs suffer from large memory requirements and computational burdens owing to their growing network structures. To address this issue, a random features approximation for online multikernel adaptive filtering is presented in this paper. Based on this efficient approximation, a novel robust MKAF based on the minimum kernel mean p-power error criterion, namely multiple random features kernel mean p-power (MRFKMP) algorithm, is therefore proposed in a fixed-dimensional random Fourier features space. The mean square steady-state performance of MRFKMP is derived for theoretical analysis in terms of excess mean square error (EMSE). In addition, an extension of MRFKMP is further developed in a recursive form for performance improvement. Monte Carlo simulations are conducted to validate the obtained theoretical results and the superiorities of the proposed algorithms from the aspects of computational efficiency, filtering accuracy, robustness to large outliers.

Kernel-target alignment based non-linear metric learning

Distance metric learning aims to learn a measure of the pairwise distance between data instances, which is essential for various machine learning algorithms and applications. However, existing linear metric learning methods based on linear transformations fail to capture nonlinear relationships between instances, while most existing kernel-based metric learning algorithms may ignore the correlation between the kernel function and the target learning task during model selection, which results in a suboptimal selection of kernels. To address the aforementioned issues, we propose a method named Kernel Alignment based Metric Learning with Random Fourier Approximation (KAMLRFA). Specifically, on one hand, we utilize the random Fourier features to approximate the shift-invariant kernel function for distance metric learning. On the other hand, we attempt to maximize the degree of agreement between the kernel function and the target learning task. In this way, distance metric learning is performed in a discriminative feature space. Compared with those kernel metric learning algorithms whose kernel functions fall into a suboptimal, KAMLRFA achieves the optimal kernel function for the target task and improves classification accuracy. We develop an efficient solution to solve the proposed optimization problem. Extensive experiments are conducted on several benchmark datasets including NUS-WIDE-LITE, USPS, MNIST, LETTER and 10 UCI datasets to verify the effectiveness of KAMLRFA compared with state-of-the-art methods.

Random fourier feature based music-speech classification

Random fourier feature based music-speech classification

Space-Efficient Feature Maps for String Alignment Kernels

String kernels are attractive data analysis tools for analyzing string data.Among them, alignment kernels are known for their high prediction accuracies in string classifications when tested in combination with SVM in various applications. However, alignment kernels have a crucial drawback in that they scale poorly due to their quadratic computation complexity in the number of input strings, which limits large-scale applications in practice. We address this need by presenting the first approximation for string alignment kernels, which we call space-efficient feature maps for edit distance with moves (SFMEDM), by leveraging a metric embedding named edit-sensitive parsing and feature maps (FMs) of random Fourier features (RFFs) for large-scale string analyses.The original FMs for RFFs consume a huge amount of memory proportional to the dimension d of input vectors and the dimension D of output vectors, which prohibits its large-scale applications.We present novel space-efficient feature maps (SFMs) of RFFs for a space reduction from O(dD) of the original FMs to O(d) of SFMs with a theoretical guarantee with respect to concentration bounds. We experimentally test SFMEDM on its ability to learn SVM for large-scale string classifications with various massive string data, and we demonstrate the superior performance of SFMEDM with respect to prediction accuracy, scalability and computation efficiency.

Random Fourier Features via Fast Surrogate Leverage Weighted Sampling

In this paper, we propose a fast surrogate leverage weighted sampling strategy to generate refined random Fourier features for kernel approximation. Compared to the current state-of-the-art method that uses the leverage weighted scheme (Li et al. 2019), our new strategy is simpler and more effective. It uses kernel alignment to guide the sampling process and it can avoid the matrix inversion operator when we compute the leverage function. Given n observations and s random features, our strategy can reduce the time complexity for sampling from O(ns2+s3) to O(ns2), while achieving comparable (or even slightly better) prediction performance when applied to kernel ridge regression (KRR). In addition, we provide theoretical guarantees on the generalization performance of our approach, and in particular characterize the number of random features required to achieve statistical guarantees in KRR. Experiments on several benchmark datasets demonstrate that our algorithm achieves comparable prediction performance and takes less time cost when compared to (Li et al. 2019).

Least Minimum Symbol Error Rate Based Post-Distortion for VLC Using Random Fourier Features

Visible light communications (VLC) has emerged as a viable supplement for high speed next-generation communication systems. However, performance of a VLC link is limited by the nonlinear characteristics of light emitting diodes (LEDs), ambient light, and inter-symbol interference (ISI). In this letter, the paradigm of least-squares based post-distortion over reproducing kernel Hilbert spaces (RKHS) is considered for a visible light communication (VLC) link. The performance-limiting impairments (viz. LED nonlinearity, and ISI), are mitigated by random Fourier features (RFF) based least minimum symbol error-rate post-distorter in RKHS, which facilitates post-distortion under finite memory-budget constrained scenarios. Performance of the proposed algorithm is tested over IEEE 802.15r1 PAN VLC channels, and the proposed algorithm is observed to converge significantly faster, whilst supporting higher data rates as compared to the existing RKHS based post-distorters.

AVBH: Asymmetric Learning to Hash with Variable Bit Encoding

Nearest neighbour search (NNS) is the core of large data retrieval. Learning to hash is an effective way to solve the problems by representing high-dimensional data into a compact binary code. However, existing learning to hash methods needs long bit encoding to ensure the accuracy of query, and long bit encoding brings large cost of storage, which severely restricts the long bit encoding in the application of big data. An asymmetric learning to hash with variable bit encoding algorithm (AVBH) is proposed to solve the problem. The AVBH hash algorithm uses two types of hash mapping functions to encode the dataset and the query set into different length bits. For datasets, the hash code frequencies of datasets after random Fourier feature encoding are statistically analysed. The hash code with high frequency is compressed into a longer coding representation, and the hash code with low frequency is compressed into a shorter coding representation. The query point is quantized to a long bit hash code and compared with the same length cascade concatenated data point. Experiments on public datasets show that the proposed algorithm effectively reduces the cost of storage and improves the accuracy of query.

Kernel Correntropy Conjugate Gradient Algorithms Based on Half-Quadratic Optimization.

As a nonlinear similarity measure defined in the kernel space, the correntropic loss (C-Loss) can address the stability issues of second-order similarity measures thanks to its ability to extract high-order statistics of data. However, the kernel adaptive filter (KAF) based on the C-Loss uses the stochastic gradient descent (SGD) method to update its weights and, thus, suffers from poor performance and a slow convergence rate. To address these issues, the conjugate gradient (CG)-based correntropy algorithm is developed by solving the combination of half-quadratic (HQ) optimization and weighted least-squares (LS) problems, generating a novel robust kernel correntropy CG (KCCG) algorithm. The proposed KCCG with less computational complexity achieves comparable performance to the kernel recursive maximum correntropy (KRMC) algorithm. To further curb the growth of the network in KCCG, the random Fourier features KCCG (RFFKCCG) algorithm is proposed by transforming the original input data into a fixed-dimensional random Fourier features space (RFFS). Since only one current error information is used in the loss function of RFFKCCG, it can provide a more efficient filter structure than the other KAFs with sparsification. The Monte Carlo simulations conducted in the prediction of synthetic and real-world chaotic time series and the regression for large-scale datasets validate the superiorities of the proposed algorithms in terms of robustness, filtering accuracy, and complexity.

Accelerate Convergence of Polarized Random Fourier Feature-Based Kernel Adaptive Filtering With Variable Forgetting Factor and Step Size

The random Fourier feature as an efficient kernel approximation method can effectively suppress the network growth of the traditional kernel-based adaptive filtering algorithm. Polarized random Fourier feature kernel least-mean-square(PRFFKLMS) remarkably improved the accuracy performance of random Fourier feature-based kernel least-mean-square algorithm and become the most representative random Fourier feature-based least-mean-square algorithm. In this paper, we studied the method that can improve the convergence speed of random Fourier feature-based least-mean-square algorithm. Based on the variable forgetting factor and variable step size strategy, three algorithm are proposed. The computational complexity of proposed algorithms are also given. The simulation results show that compared with PRFFKLMS algorithm, the convergence speed of the proposed algorithm is significantly improved.

A Neural Network Approach for Online Nonlinear Neyman-Pearson Classification

We propose a novel Neyman-Pearson (NP) classifier that is both online and nonlinear as the first time in the literature. The proposed classifier operates on a binary labeled data stream in an online manner, and maximizes the detection power about a user-specified and controllable false positive rate. Our NP classifier is a single hidden layer feedforward neural network (SLFN), which is initialized with random Fourier features (RFFs) to construct the kernel space of the radial basis function at its hidden layer with sinusoidal activation. Not only does this use of RFFs provide an excellent initialization with great nonlinear modeling capability, but it also exponentially reduces the parameter complexity and compactifies the network to mitigate overfitting while improving the processing efficiency substantially. We sequentially learn the SLFN with stochastic gradient descent updates based on a Lagrangian NP objective. As a result, we obtain an expedited online adaptation and powerful nonlinear Neyman-Pearson modeling. Our algorithm is appropriate for large scale data applications and provides a decent false positive rate controllability with real time processing since it only has O(N) computational and O(1) space complexity (N: number of data instances). In our extensive set of experiments on several real datasets, our algorithm is highly superior over the competing state-of-the-art techniques, either by outperforming in terms of the NP classification objective with a comparable computational as well as space complexity or by achieving a comparable performance with significantly lower complexity.

Efficient Kernel Cook's Distance for Remote Sensing Anomalous Change Detection

Detecting anomalous changes in remote sensing images is a challenging problem, where many approaches and techniques have been presented so far. We rely on the standard field of multivariate statistics of diagnostic measures, which are concerned about the characterization of distributions, detection of anomalies, extreme events, and changes. One useful tool to detect multivariate anomalies is the celebrated Cook's distance. Instead of assuming a linear relationship, we present a novel kernelized version of the Cook's distance to address anomalous change detection in remote sensing images. Due to the large computational burden involved in the direct kernelization, and the lack of out-of-sample formulas, we introduce and compare both random Fourier features and Nystrom implementations to approximate the solution. We study the kernel Cook's distance for anomalous change detection in a chronochrome scheme, where the anomalousness indicator comes from evaluating the statistical leverage of the residuals of regressors between time acquisitions. We illustrate the performance of all algorithms in a representative number of multispectral and very high resolution satellite images involving changes due to droughts, urbanization, wildfires, and floods. Very good results and computational efficiency confirm the validity of the approach.

An Efficient Nonlinear Dichotomous Coordinate Descent Adaptive Algorithm Based on Random Fourier Features

The auxiliary normal equation is proposed to construct an incremental update (IU) system in which the increment of weight vector rather than the weight itself is optimized at each iteration, which however, can only deal with problems under linearity assumption. In this letter, we establish the IU system in a fixed-dimension feature space induced by the random Fourier features mapping, enabling it to address nonlinear issues efficiently. Then, to obtain the solution vector to the IU system and overcome high complexity and instability incurred by matrix inversion, a dichotomous coordinate descent (DCD) method is employed to propose an efficient random Fourier features kernel dichotomous coordinate descent (RFFKDCD) algorithm. The proposed RFFKDCD algorithm can tackle nonlinear issues efficiently while avoiding high computational complexity existing widely in kernel-based algorithms. Simulations on both synthetic and real-world examples show that the proposed RFFKDCD algorithm outperforms the recursive least squares-based algorithms in terms of computational load.

Exponential inequalities for dependent V-statistics via random Fourier features

We establish exponential inequalities for a class of V-statistics under strong mixing conditions. Our theory is developed via a novel kernel expansion based on random Fourier features and the use of a probabilistic method. This type of expansion is new and useful for handling many notorious classes of kernels.

Complex-Valued Adaptive Filtering Based on the Random Fourier Features Method

The complex kernel adaptive filters (CKAFs) developed in complex reproducing kernel Hilbert space (RKHS) improve the performance of complex linear adaptive filters, but result in large burdens of computation and memory. To address these issues, a novel complex random Fourier features mapping (CRFFM) is proposed to approximate the kernel-induced mapping by applying the complexification of real RKHSs, and thus projects the complex-valued data from the original data space into the fixed-dimensional complex random Fourier features space (RFFS). To combat complex-valued non-Gaussian noises, a complex Cauchy loss function is presented, and the complex random Fourier features recursive complex Cauchy (CRFFRCC) algorithm is therefore proposed by the combination of CRFFM and the stochastic recursive method. The proposed CRFFRCC with a linear filter structure can reduce the computational and space complexities of CKAFs, significantly. Monte Carlo simulations conducted in the complex-valued nonlinear channel equalization validate the superiorities of CRFFRCC.

Nonlinear Distribution Regression for Remote Sensing Applications

In many remote sensing applications, one wants to estimate variables or parameters of interest from observations. When the target variable is available at a resolution that matches the remote sensing observations, standard algorithms, such as neural networks, random forests, or the Gaussian processes, are readily available to relate the two. However, we often encounter situations where the target variable is only available at the group level, i.e., collectively associated with a number of remotely sensed observations. This problem setting is known in statistics and machine learning as multiple instance learning (MIL) or distribution regression (DR). This article introduces a nonlinear (kernel-based) method for DR that solves the previous problems without making any assumption on the statistics of the grouped data. The presented formulation considers distribution embeddings in reproducing kernel Hilbert spaces and performs standard least squares regression with the empirical means therein. A flexible version to deal with multisource data of different dimensionality and sample sizes is also presented and evaluated. It allows working with the native spatial resolution of each sensor, avoiding the need for matchup procedures. Noting the large computational cost of the approach, we introduce an efficient version via random Fourier features to cope with millions of points and groups. Real experiments involve the Soil Moisture Active Passive (SMAP) vegetation optical depth (VOD) data for the estimation of crop production in the U.S. Corn Belt and the Moderate Resolution Imaging Spectroradiometer (MODIS) and Multi-angle Imaging SpectroRadiometer (MISR) reflectances for the estimation of aerosol optical depth (AOD). An exhaustive empirical evaluation of the method is done against naive (linear and nonlinear) approaches based on input-space means as well as previously presented methods for MIL. We provide source code of our methods in http://isp.uv.es/code/dr.html .

Feature uncertainty bounds for explicit feature maps and large robust nonlinear SVM classifiers

We consider the binary classification problem when data are large and subject to unknown but bounded uncertainties. We address the problem by formulating the nonlinear support vector machine training problem with robust optimization. To do so, we analyze and propose two bounding schemes for uncertainties associated to random approximate features in low dimensional spaces. The proposed bound calculations are based on Random Fourier Features and the Nystrom methods. Numerical experiments are conducted to illustrate the benefit of the technique. We also emphasize the decomposable structure of the proposed robust nonlinear formulation that allows the use of efficient stochastic approximation techniques when datasets are large.

The Cauchy Conjugate Gradient Algorithm with Random Fourier Features

Random Fourier mapping (RFM) in kernel adaptive filters (KAFs) provides an efficient method to curb the linear growth of the dictionary by projecting the original input data into a finite-dimensional space. The commonly used measure in RFM-based KAFs is the minimum mean square error (MMSE), which causes performance deterioration in the presence of non-Gaussian noises. To address this issue, the minimum Cauchy loss (MCL) criterion has been successfully applied for combating non-Gaussian noises in KAFs. However, these KAFs using the well-known stochastic gradient descent (SGD) optimization method may suffer from slow convergence rate and low filtering accuracy. To this end, we propose a novel robust random Fourier features Cauchy conjugate gradient (RFFCCG) algorithm using the conjugate gradient (CG) optimization method in this paper. The proposed RFFCCG algorithm with low complexity can achieve better filtering performance than the KAFs with sparsification, such as the kernel recursive maximum correntropy algorithm with novelty criterion (KRMC-NC), in stationary and non-stationary environments. Monte Carlo simulations conducted in the time-series prediction and nonlinear system identification confirm the superiorities of the proposed algorithm.