Fourier Features Research Articles

Zero-Attracting Kernel Maximum Versoria Criterion Algorithm for Nonlinear Sparse System Identification

Sparsity-induced kernel adaptive filters have emerged as a promising candidate for a nonlinear sparse system identification (SSI) problem. The existing zero-attracting kernel least mean square (ZA-KLMS) algorithm relies on minimum mean square error criterion, which considers only second order statistics of error, thereby resulting in suboptimal performance in the presence of non-Gaussian/impulsive distortions. In this letter, we propose a novel random Fourier features (RFF) based ZA kernel maximum Versoria criterion (ZA-KMVC) algorithm, and their variants, which are robust for nonlinear SSI in the presence of non-Gaussian distortions over both stationary and time-varying environments. Furthermore, the mean-square convergence analysis of the proposed RFF-ZA-KMVC algorithm is performed. It has been observed from the simulation results that the proposed algorithm delivers better convergence performance as compared to the existing state-of-art approaches.

Comparative Analytical Study of SCMA Detection Methods for PA Nonlinearity Mitigation.

Non-orthogonal multiple access (NOMA) has emerged as a promising technology that allows for multiplexing several users over limited time-frequency resources. Among existing NOMA methods, sparse code multiple access (SCMA) is especially attractive; not only for its coding gain using suitable codebook design methodologies, but also for the guarantee of optimal detection using message passing algorithm (MPA). Despite SCMA’s benefits, the bit error rate (BER) performance of SCMA systems is known to degrade due to nonlinear power amplifiers at the transmitter. To mitigate this degradation, two types of detectors have recently emerged, namely, the Bussgang-based approaches and the reproducing kernel Hilbert space (RKHS)-based approaches. This paper presents analytical results on the error-floor of the Bussgang-based MPA, and compares it with a universally optimal RKHS-based MPA using random Fourier features (RFF). Although the Bussgang-based MPA is computationally simpler, it attains a higher BER floor compared to its RKHS-based counterpart. This error floor and the BER’s performance gap are quantified analytically and validated via computer simulations.

Slim-RFFNet: Slim deep convolution random Fourier feature network for image classification

The standard kernel method is computationally expensive because it needs to store and compute the inverse of the Gram matrix. Furthermore, the classification accuracy of the single kernel method is not effective or efficient as the method’s ability to extract features. The random Fourier feature method establishes a connection between the kernel method and deep learning to solve the above problems. In this paper, we propose a novel slim deep random Fourier feature network named Slim-RFFNet, which introduces convolution into kernel learning. We use the hierarchical strategy and skip connection to construct a deep network structure and lighten the model by using quantization. Experiments conducted on classification benchmarks MNIST and CIFAR10 demonstrate that the proposed Slim-RFFNet significantly outperforms current state-of-the-art deep kernel learning methods. Our algorithm also achieves a trade-off between accuracy and latency. The proposed network can be applied to resource-constrained embedded AI devices. The experimental results on the edge computing system show that our algorithm has a small memory footprint and fast inference speed on small edge devices, and thus meets the requirements for practical applications.

A random matrix analysis of random Fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent* *This article is an updated version of: Liao Z, Couillet R and Mahoney M W 2020 A random matrix analysis of random Fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent Advances in

This article characterizes the exact asymptotics of random Fourier feature (RFF) regression, in the realistic setting where the number of data samples n, their dimension p, and the dimension of feature space N are all large and comparable. In this regime, the random RFF Gram matrix no longer converges to the well-known limiting Gaussian kernel matrix (as it does when N → ∞ alone), but it still has a tractable behavior that is captured by our analysis. This analysis also provides accurate estimates of training and test regression errors for large n, p, N. Based on these estimates, a precise characterization of two qualitatively different phases of learning, including the phase transition between them, is provided; and the corresponding double descent test error curve is derived from this phase transition behavior. These results do not depend on strong assumptions on the data distribution, and they perfectly match empirical results on real-world data sets.

Time-Frequency Sparse Reconstruction of Non-Uniform Sampling for Non-Stationary Signal

We address the problem of the reconstruction of time-frequency characteristics of sparse multi-band signals by using the discrete multi-coset sampling (DMCS) model. In this article, the signal is characterized by complicated time-variable components. According to the feature of the short-time Fourier transform (STFT) analysis method, we obtain the rewritten matrix form of the discrete STFT. Then, an analysis method of the multi-coset sliding window (MCSW) is proposed, and the discrete multi-coset sampling sequence is locally windowed. The sparse signal reconstruction algorithm is used to obtain the optimal time-frequency reconstruction value of the original signal. Numerical simulations show the feasibility of this method. We simulate the effects of measurement noise, sampling rate, and time-frequency analysis parameters on the reconstruction accuracy. Our method can carry out time-frequency reconstruction for undersampled signals obtained from multi-coset sampling to ensure the time-varying feature of the original signals, which can well highlight the characteristics of signals. The method is of great significance to the research and development of the sub-Nyquist sampling technique.

Wind field reconstruction with adaptive random Fourier features

We investigate the use of spatial interpolation methods for reconstructing the horizontal near-surface wind field given a sparse set of measurements. In particular, random Fourier features is compared with a set of benchmark methods including kriging and inverse distance weighting. Random Fourier features is a linear model approximating the velocity field, with randomly sampled frequencies and amplitudes trained to minimize a loss function. We include a physically motivated divergence penalty , as well as a penalty on the Sobolev norm of . We derive a bound on the generalization error and a sampling density that minimizes the bound. We then devise an adaptive Metropolis–Hastings algorithm for sampling the frequencies of the optimal distribution. In our experiments, our random Fourier features model outperforms the benchmark models.

Nazgul: A statistical approach to gamma-ray burst localization

Context. Gamma-ray bursts (GRBs) can be located via arrival time signal triangulation using gamma-ray detectors in orbit throughout the solar system. The classical approach based on cross-correlations of binned light curves ignores the Poisson nature of the time series data, and it is unable to model the full complexity of the problem. Aims. We aim to present a statistically proper and robust GRB timing and triangulation algorithm as a modern update to the original procedures used for the interplanetary network. Methods. A hierarchical Bayesian forward model for the unknown temporal signal evolution is learned via random Fourier features and fitted to each detector’s time series data with time differences that correspond to the position GRBs on the sky via the appropriate Poisson likelihood. Results. Our novel method can robustly estimate the position of a GRB as verified via simulations. The uncertainties generated by the method are robust and in many cases more precise compared to the classical method. Thus, we have a method that can become a valuable tool for gravitational wave follow-up.

Sketching Data Sets for Large-Scale Learning: Keeping only what you need

Big data can be a blessing: with very large training data sets it becomes possible to perform complex learning tasks with unprecedented accuracy. Yet, this improved performance comes at the price of enormous computational challenges. Thus, one may wonder: Is it possible to leverage the information content of huge data sets while keeping computational resources under control? Can this also help solve some of the privacy issues raised by large-scale learning? This is the ambition of compressive learning, where the data set is massively compressed before learning. Here, a "sketch" is first constructed by computing carefully chosen nonlinear random features [e.g., random Fourier (RF) features] and averaging them over the whole data set. Parameters are then learned from the sketch, without access to the original data set. This article surveys the current state of the art in compressive learning, including the main concepts and algorithms, their connections with established signal processing methods, existing theoretical guarantees on both information preservation and privacy preservation, and important open problems. For an extended version of this article that contains additional references and more in-depth discussions on a variety of topics, see [1].

Online robust echo state broad learning system

Recurrent broad learning system (RBLS) is an effective learning method for processing sequential data. By replacing the enhancement nodes of the broad learning system with recurrent structure, RBLS obtains the capacity to capture the dynamic characteristics of time series data. However, RBLS is derived under the minimum mean square error (MMSE) criterion, which is sensitive to outliers. Moreover, RBLS is insufficient for online sequential learning. To address these limitations, we propose a novel online robust echo state structure based RBLS (OR-ESBLS). In OR-ESBLS, kernel recursive maximum correntropy (KRMC) is introduced to both enhance the robustness and discover the nonlinear characteristics of feature nodes in an online manner. To reduce the heavy computational requirements caused by the kernel method, a Quasi-Monte Carlo (QMC) based Random Fourier Feature (RFF) is utilized for kernel approximation. Furthermore, we adopt the randomized sparse reservoir as the enhancement nodes of RBLS, which can much more efficiently capture dynamic information of the data in the sequential learning setting. Experiments on both synthetic and real-world datasets are reported. The results show that the proposed OR-ESBLS can provide superior performance in online sequential time series prediction.

Dimensionality reduction in the context of dynamic social media data streams

In recent years social media became an important part of everyday life for many people. A big challenge of social media is, to find posts, that are interesting for the user. Many social networks like Twitter handle this problem with so-called hashtags. A user can label his own Tweet (post) with a hashtag, while other users can search for posts containing a specified hashtag. But what about finding posts which are not labeled by the creator? We provide a way of completing hashtags for unlabeled posts using classification on a novel real-world Twitter data stream. New posts will be created every second, thus this context fits perfectly for non-stationary data analysis. Our goal is to show, how labels (hashtags) of social media posts can be predicted by stream classifiers. In particular, we employ random projection (RP) as a preprocessing step in calculating streaming models. Also, we provide a novel real-world data set for streaming analysis called NSDQ with a comprehensive data description. We show that this dataset is a real challenge for state-of-the-art stream classifiers. While RP has been widely used and evaluated in stationary data analysis scenarios, non-stationary environments are not well analyzed. In this paper, we provide a use case of RP on real-world streaming data, especially on NSDQ dataset. We discuss why RP can be used in this scenario and how it can handle stream-specific situations like concept drift. We also provide experiments with RP on streaming data, using state-of-the-art stream classifiers like adaptive random forest and concept drift detectors. Additionally, we experimentally evaluate an online principal component analysis (PCA) approach in the same fashion as we do for RP. To obtain higher dimensional synthetic streams, we use random Fourier features (RFF) in an online manner which allows us, to increase the number of dimensions of low dimensional streams.

Random Fourier feature kernel recursive maximum mixture correntropy algorithm for online time series prediction

In the paper, a novel kernel recursive least-squares (KRLS) algorithm named random Fourier feature kernel recursive maximum mixture correntropy (RFF-RMMC) algorithm is proposed, which improves the prediction efficiency and robustness of the KRLS algorithm. Random Fourier feature (RFF) method as well as maximum mixture correntropy criterion (MMCC) are combined and applied into KRLS algorithm afterwards. Using RFF to approximate the kernel function in KRLS with a fixed cost can greatly reduce the computational complexity and simultaneously improve the prediction efficiency. In addition, the MMCC maintains the robustness like the maximum correntropy criterion (MCC). More importantly, it can enhance the accuracy of the similarity measurement between predicted and true values by more flexible parameter settings, and then make up for the loss of prediction accuracy caused by RFF to a certain extent. The performance of the RFF-RMMC algorithm for online time series prediction is verified by the simulation results based on three datasets.

Efficient batch process monitoring based on random nonlinear feature analysis

AbstractEfficient batch process monitoring is a challenging task because of the large data scale, the strong process nonlinearity, and the complicated local behaviours. To handle this issue, this paper presents a random nonlinear feature analysis method, called two‐level localized ensemble random Fourier feature analysis (TLERFFA), for effective batch process fault detection. Considering the large data scale of the nonlinear batch process, the random Fourier feature analysis (RFFA) is introduced to develop the statistical model, which is more efficient than the traditional kernel modelling method in terms of computation loads. Furthermore, for overcoming the model uncertainty resulting from the random mapping weights, the ensemble learning based on Bayesian inference is used to integrate multiply RFFA models with different weight settings. To further mine the rich local information of batch process data, a two‐level localization strategy is designed from the perspectives of the variables and the statistics. At the first level, the monitored variables are divided into several sub‐groups according to their maximal information coefficients, while at the second level, the local monitoring statistics are constructed based on the temporal–spatial neighbour analysis. Lastly, one case study on the benchmark process is given to demonstrate the effectiveness of the proposed methods.

Classification of Officers’ Driving Situations Based on Eye-Tracking and Driver Performance Measures

Motor vehicle crashes are a leading cause of police officers' deaths in the line of duty. These crashes are mainly attributed to officers' use of in-vehicle technologies while driving, distraction, fatigue, and high-speed driving conditions. The objective of this study is to classify officers' driving situations using a combination of driver behavior and eye-tracking measures. The study compared three algorithms, including random forest (RF), support vector machine (SVM), and random Fourier features (RFF) to classify officers' driving situations (i.e., normal vs. pursuit driving) and in-vehicle technology use. The results suggested that driver behavior measures, combined with RF or SVM methods, are most promising for classifying officers' driving condition (accuracy of about 90%). However, it might be more efficient to apply RFF with driver behavior measures to classify officers' use of in-vehicle technologies while driving due to the time cost reduction of RFF as compared to SVM and RF algorithms. The findings can be applied to improve future police vehicles, training protocols, and to provide adaptive technology solutions to reduce officers' driving distraction and workload.

Performance Analysis of Random Fourier Features-Based Unsupervised Multistage-Clustering for VLC

Visible light communication (VLC) has emerged as a secure, cost-effective, and green supplement to the existing radio frequency (RF) communication systems. However, the performance of a typical VLC link is degraded by the following factors: (a) inter-symbol interference (ISI), which arises due to limited modulation bandwidth of light-emitting diode (LED), (b) the dispersion due to multi-path reflections, and (c) nonlinear transfer characteristics of LED. To mitigate ISI and LED nonlinearity, existing blind post-distorters, like the Volterra modified cascaded multimodulus algorithm (MCMMA) and the Hammerstein improved multistage clustering (HIMSC) approaches, experience degradations from modeling errors due to abrupt polynomial-series truncation. To mitigate these degradations, we propose a novel random Fourier features (RFF) based improved multi-stage clustering (RFF-IMSC) algorithm for blind post-distortion. Simulations performed over standardized IEEE 802.15 PAN VLC channels indicate that the proposed RFF-RIMSC post-distorter delivers improved performance as compared to the existing blind post-distorters. Also, analytical guarantees are presented and error-rate performance are analyzed for the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon $ </tex-math></inline-formula> -optimality of this approach with respect to supervised loss-functions.

Random Fourier feature-based fuzzy clustering with [formula omitted]-Laplacian regularization

Random feature is one successful technique to approximate traditional kernel functions, and the random feature-based fuzzy clustering has been proved to be effective and efficient for handling non-linear data. However, the existing random feature-based fuzzy clustering methods fail to consider the locality information hidden in original input data. From some perspective, the membership obtained by fuzzy clustering can be seen as the encoding results of data. Thus, constraining the relationships between membership degrees to be consistent with that of data is beneficial to improve clustering performance. To this end, we propose a novel random Fourier feature-based fuzzy clustering method (pLRFCM) in this paper. The random Fourier feature is used to approximate Gaussian kernels in this method, and the fuzzy clustering is performed in the feature space. More importantly, the p-Laplacian regularization is conducted on the membership matrix to preserve the local structures of original data into the clustering results, to guarantee good partition of data. The maximum-entropy technique is also utilized to fine-tune the weights of features automatically during the process of clustering, so as to further promote the performance of clustering. In the experiments on four synthetic non-linear datasets and eight real-world datasets, pLRFCM outperforms several classical and state-of-the-art fuzzy clustering methods.

On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction–diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at https://github.com/PredictiveIntelligenceLab/MultiscalePINNs.

SDCRKL-GP: Scalable deep convolutional random kernel learning in gaussian process for image recognition

Deep convolutional neural networks have shown great potential in image recognition tasks. However, the fact that the mechanism of deep learning is difficult to explain hinders its development. It involves a large amount of parameter learning, which results in high computational complexity. Moreover, deep convolutional neural networks are often limited by overfitting in regimes in which the number of training samples is limited. Conversely, kernel learning methods have a clear mathematical theory, fewer parameters, and can contend with small sample sizes; however, they are not able to handle high-dimensional data, e.g., images. It is important to achieve a performance and complexity trade-off in complicated tasks. In this paper, we propose a novel scalable deep convolutional random kernel learning in Gaussian process architecture called SDCRKL-GP, which is characterized by excellent performance and low complexity. First, we successfully incorporated the deep convolutional architecture into kernel learning by implementing the random Fourier feature transform for Gaussian processes, which can effectively capture hierarchical and local image-level features. This approach enabled the kernel method to effectively handle image processing problems. Second, we optimized the parameters of deep convolutional filters and Gaussian kernels by stochastic variational inference. Then, we derived the lower variational bound of the marginal likelihood. Finally, we explored the model architecture design space selection method to determine the appropriate network architecture for different datasets. The design space consists of the number of layers, the channels per layer, and so on. Different design space selections improved the scalability of the SDCRKL-GP architecture. We evaluated SDCRKL-GP on the MNIST, FMNIST, CIFAR10, and CALTECH4 benchmark datasets. Taking MNIST as an example, the error rate of classification is 0.60%, and the number of parameters, number of computations and memory access cost of the architecture are 19.088k, 0.984M, and 1.057M, respectively. The experimental results verified that the proposed SDCRKL-GP method outperforms several state-of-the-art algorithms in both accuracy and speed in image recognition tasks. The code is available at https://github.com/w-tingting/deep-rff-pytorch.

Breaking the waves: asymmetric random periodic features for low-bitrate kernel machines

Abstract Many signal processing and machine learning applications are built from evaluating a kernel on pairs of signals, e.g., to assess the similarity of an incoming query to a database of known signals. This nonlinear evaluation can be simplified to a linear inner product of the random Fourier features (RFFs) of those signals: random projections followed by a periodic map, the complex exponential. It is known that a simple quantization of those features (corresponding to replacing the complex exponential by a different periodic map that takes binary values, which is appealing for their transmission and storage) distorts the approximated kernel, which may be undesirable in practice. Our take-home message is that when the features of only one of the two signals are quantized, the original kernel is recovered without distortion; its practical interest appears in several cases where the kernel evaluations are asymmetric by nature, such as a client-server scheme. Concretely, we introduce the general framework of asymmetric random periodic features, where the two signals of interest are observed through random periodic features—random projections followed by a general periodic map, which is allowed to be different for both signals. We derive the influence of those periodic maps on the approximated kernel and prove uniform probabilistic error bounds holding for all pair of signals picked in an infinite low-complexity set. Interestingly, our results allow the periodic maps to be discontinuous, thanks to a new mathematical tool, i.e., the mean Lipschitz smoothness. We then apply this generic framework to semi-quantized kernel machines (where only one of the signals has quantized features and the other has classical RFFs), for which we show theoretically that the approximated kernel remains unchanged (with the associated error bound) and confirm the power of the approach with numerical simulations.

Least minimum symbol error rate based post-distortion for adaptive mobile VLC transmission with receiver selection

In the context of beyond-5G indoor communication systems, visible light communication (VLC) has emerged as a viable supplement for existing RF-based systems and as an enabler for high data-rate communications. However, the existing indoor VLC systems are limited by detrimental outages caused by fluctuations in the VLC channel-gain due to user-mobility. Furthermore, the nonlinear characteristics of the light-emitting diode (LED) degrade the performance of VLC systems in the high-power regime by warping the input constellation. Additional performance-limits are introduced by inter-symbol interference (ISI) due to finite modulation-bandwidth of LEDs, and reflections from walls. In this paper, a random Fourier feature (RFF) based post-distorter is considered for mitigating the LED nonlinearity, and relevant expressions for the signal to noise ratio (SNR) are derived for a direct current biased optical orthogonal frequency division multiplexing (DCO-OFDM) system. Based on the derived expressions for SNR, the effects of user-mobility and ISI are mitigated by a DCO-OFDM based adaptive VLC transmission technique, which varies the transmission-rate/modulation-order under a specified error-rate constraint. Simulations are presented over channels obtained by ray-tracing, which indicates that the proposed algorithm achieves superior data-rates with a significantly lower error-rate.

Complex multi-kernel random Fourier adaptive algorithms under the complex kernel risk-sensitive p-power loss

The complex least mean square (CLMS) adaptive algorithm based on the minimum mean square error (MSE) criterion has been widely used for linear applications with Gaussian noises in complex domain. However, the MSE criterion suffers from performance degeneration in the presence of non-Gaussian noise. To address this issue, a novel complex kernel risk-sensitive p-power loss (CKRSP-L) criterion is first constructed to combat non-Gaussian noises. Then, based on the constructed CKRSP-L criterion, a novel complex kernel risk-sensitive p-power (CKRSP) algorithm is proposed to provide robustness to non-Gaussian noises and performance improvement for linear systems, simultaneously. Further, to extend the CKRSP algorithm into nonlinear systems, a novel complex multi-kernel random Fourier mapping (CMRFM) is proposed to transform the original input data into a complex multi-kernel random Fourier features space (CMRFFS), and thus another novel complex multi-kernel random Fourier kernel risk-sensitive p-power (CMRFKRSP) algorithm is presented for nonlinear applications in complex domain. Finally, the steady-state excess mean square errors (SEMSEs) of CKRSP and CMRFKRSP are also calculated for theoretical analysis of performance. Monte Carlo simulations conducted in different noise environments validate the correctness of the obtained SEMSEs and performance advantages of CKRSP and CMRFKRSP.