Nyström-aware approximations for matrix-based Rényi's entropy.

Information-Theoretic Learning (ITL) is one of the new methods gaining popularity used for adaptive signal processing learning algorithms and has many advantages compared to traditional method which minimizes the mean square error (MSE). Previously [12], we described a method based on the backpropagation algorithm to train a type of neural network called the multi-layer perceptron (MLP) using Information Theoretic Learning (ITL) techniques. Our method was developed to train MLPs by utilizing the minimum error entropy (MEE) of the error samples. The MSE is a second order statistic whereas the MEE uses the probability density function of the error samples. Therefore, the MEE technique uses higher order statistical information from the error samples to adapt the weights of the neural network. When the error distribution is non-gaussian, higher order statistical information can lead to faster training and smaller residual training error. The Probability Density Function (PDF) estimation using the Parzen window could affect the accuracy of the Back-Propagation training. In this paper, we investigate the effects of the Parzen Window estimator on the efficacy of the ITL training using Renyi's Entropy and Shannon's Entropy. Using different estimators and simulations, we compare MLP using the typical backpropagation algorithm (using MSE and cross-entropy) and also one using ITL methods in terms of convergence speed of the weights, PDF estimator and the residual error. We use standard data sets (like the MNIST handwriting data set available on the Internet) to train and test the MLP using all these methods. Simulation results compare the prediction accuracy of the three different types of backpropagation algorithms (MSE, Shannon's cross-entropy, Renyi's quadratic entropy) in the paper.

Information potential for some probability density functions

On reduced rank nonnegative matrix factorization for symmetric nonnegative matrices

Wind power forecasting (WPF) provides important inputs to power system operators and electricity market participants. It is therefore not surprising that WPF has attracted increasing interest within the electric power industry. In this report, we document our research on improving statistical WPF algorithms for point, uncertainty, and ramp forecasting. Below, we provide a brief introduction to the research presented in the following chapters. For a detailed overview of the state-of-the-art in wind power forecasting, we refer to [1]. Our related work on the application of WPF in operational decisions is documented in [2]. Point forecasts of wind power are highly dependent on the training criteria used in the statistical algorithms that are used to convert weather forecasts and observational data to a power forecast. In Chapter 2, we explore the application of information theoretic learning (ITL) as opposed to the classical minimum square error (MSE) criterion for point forecasting. In contrast to the MSE criterion, ITL criteria do not assume a Gaussian distribution of the forecasting errors. We investigate to what extent ITL criteria yield better results. In addition, we analyze time-adaptive training algorithms and how they enable WPF algorithms to cope with non-stationary data and, thus, to adapt to new situations without requiring additional offline training of the model. We test the new point forecasting algorithms on two wind farms located in the U.S. Midwest. Although there have been advancements in deterministic WPF, a single-valued forecast cannot provide information on the dispersion of observations around the predicted value. We argue that it is essential to generate, together with (or as an alternative to) point forecasts, a representation of the wind power uncertainty. Wind power uncertainty representation can take the form of probabilistic forecasts (e.g., probability density function, quantiles), risk indices (e.g., prediction risk index) or scenarios (with spatial and/or temporal dependence). Statistical approaches to uncertainty forecasting basically consist of estimating the uncertainty based on observed forecasting errors. Quantile regression (QR) is currently a commonly used approach in uncertainty forecasting. In Chapter 3, we propose new statistical approaches to the uncertainty estimation problem by employing kernel density forecast (KDF) methods. We use two estimators in both offline and time-adaptive modes, namely, the Nadaraya-Watson (NW) and Quantilecopula (QC) estimators. We conduct detailed tests of the new approaches using QR as a benchmark. One of the major issues in wind power generation are sudden and large changes of wind power output over a short period of time, namely ramping events. In Chapter 4, we perform a comparative study of existing definitions and methodologies for ramp forecasting. We also introduce a new probabilistic method for ramp event detection. The method starts with a stochastic algorithm that generates wind power scenarios, which are passed through a high-pass filter for ramp detection and estimation of the likelihood of ramp events to happen. The report is organized as follows: Chapter 2 presents the results of the application of ITL training criteria to deterministic WPF; Chapter 3 reports the study on probabilistic WPF, including new contributions to wind power uncertainty forecasting; Chapter 4 presents a new method to predict and visualize ramp events, comparing it with state-of-the-art methodologies; Chapter 5 briefly summarizes the main findings and contributions of this report.

Semidefinite feasibility problems arise in many areas of operations research. The abstract form of these problems can be described as finding a point in a nonempty bounded convex body Γ in the cone of symmetric positive semidefinite matrices. Assume that Γ is defined by an oracle, which for any given m × m symmetric positive semidefinite matrix Ŷ either confirms that Ŷ ∈ Γ or returns a cut, i.e., a symmetric matrix A such that Γ is in the half-space {Y : A · Y ≤ A · Ŷ}. We study an analytic center cutting plane algorithm for this problem. At each iteration, the algorithm computes an approximate analytic center of a working set defined by the cutting plane system generated in the previous iterations. If this approximate analytic center is a solution, then the algorithm terminates; otherwise the new cutting plane returned by the oracle is added into the system. As the number of iterations increases, the working set shrinks and the algorithm eventually finds a solution to the problem. All iterates generated by the algorithm are positive definite matrices. The algorithm has a worst-case complexity of O*(m3/ε2) on the total number of cuts to be used, where ε is the maximum radius of a ball contained by Γ.

In this work, a measure of similarity based on the matching of multivariate probability density functions (PDFs) is proposed. In consonance with the information theoretic learning (ITL) framework, the affinity comparison between the joint PDFs is performed using a quadratic distance, estimated with the aid of the Parzen window method with Gaussian kernels. The motivation underlying this proposal is to introduce a criterion capable of quantifying, to a significant extent, the statistical dependence present on information sources endowed with temporal and/or spatial structure, like audio, images and coded data. The measure is analyzed and compared with the canonical ITL-based approach — correntropy — for a set of blind equalization scenarios. The comparison includes elements like surface analysis, performance comparison in terms of bit error rate and a qualitative discussion concerning image processing. It is also important to remark that the study includes the application of two computational intelligence paradigms: extreme learning machines and differential evolution. The results indicate that the proposal can be, in some scenarios, a more informative formulation than correntropy.

Non-linear equalization techniques using decision feedback structure are highly demanded for cancellation of intersymbol interferences occurred in severe channel environments. In this paper decision feedback structure is applied to the linear blind equalizer algorithm that is based on information theoretic learning and a randomly generated symbol set. At the decision feedback equalizer (DFE) the random symbols are generated to have the same probability density function (PDF) as that of the transmitted symbols. By minimizing difference between the PDF of blind DFE output and that of randomly generated symbols, the proposed DFE algorithm produces equalized output signal. From the simulation results, the proposed method has shown enhanced convergence and error performance compared to its linear counterpart. Key Words : Decision feedback, Blind equalization, PDF, Random symbols, ITL. * 교신저자 : 김남용(namyong@kangwon.ac.kr)접수일 11년 11월 02일 수정일 (1차 11년 12월 26일, 2차 11년 12월 29일) 게재확정일 12년 01월 05일

A fundamental result in the theory of spatial stiffness matrices is Loncaric's normal form. When a spatial stiffness matrix is described in an appropriate coordinate frame, it will have a particularly simple structure. In this form the 3 /spl times/ 3 off-diagonal blocks of the stiffness matrix are diagonal. It has been shown that generically, a spatial stiffness matrix call be written in normal form. For example, it is fairly well known that this is possible for any positive definite spatial stiffness matrix. In this article, it is shown that any symmetric positive semi-definite matrix can also be written in normal form. As an application this result is used to design a compact parallel compliance mechanism with a prescribed positive semidefinite spatial stiffness matrix.

Distance metric learning (DML) is an effective similarity learning tool to learn a distance function from examples to enhance the model performance in applications of classification, regression, and ranking, etc. Most DML algorithms need to learn a Mahalanobis matrix, a positive semidefinite matrix that scales quadratically with the number of dimensions of input data. This brings huge computational cost in the learning procedure, and makes all proposed algorithms infeasible for extremely high-dimensional data even with the low-rank approximation. Differently, in this paper, we take advantage of the power of parallel computation and propose a novel distributed distance metric learning algorithm based on a state-of-the-art DML algorithm, Information-Theoretic Metric Learning (ITML).More specifically, we utilize the property that each positive semidefinite matrix can be decomposed into a combination of rank-one and trace-one matrices and convert the original sequential training procedure into a parallel one. In most cases, the communication demands of the proposed method are also reduced from O(d2) to O(cd), where d is the number of dimensions of the data and c is the number of constraints in DML and can be smaller than d by appropriate selection. Moreover importantly, we present a rigorous theoretical analysis to upper bound the Bregman divergence between the sequential algorithm and the parallel algorithm, which guarantees the correctness and performance of the proposed algorithm. Our experiments on datasets with O(105) features demonstrate the competitive scalability and the performance compared with the original ITML algorithm.

Recently a Hybrid Carrier (HC) scheme based on Weighted-type Fractional Fourier Transform (WFRFT) was proposed and developed, which contains Single Carrier (SC) and Multi-Carrier (MC) synergetic transmission. The wide interest is primarily due to its appealing characteristics, such as the robust performances in different types of selective fading channels and a great deal of potential for secure communications. According to the literatures, the HC signal and SC or MC signal probability distributions are different. In particular, some benefits of this HC scheme are brought by the quasi-Gaussian distribution of WFRFT signals. However, until now researchers have only presented statistic properties through computer simulations, and the accurate expressions of signals are not derived yet. In this paper, we derive the accurate and rigorously established closed-form expressions of Probability Density Function (PDF) of WFRFT signal real and imaginary parts with a large number of QPSK subcarriers, and this PDF can describe the behavior of data modulated by WFRFT, avoiding the complex computation for extensive computer simulations. Furthermore, the components of PDF expression are described and analyzed, and it is revealed that the tendency of signal quasi-Gaussian changes with the increasing of the parameter α (α in (0, 1]). To validate the analytical results, extensive simulations have been conducted, showing a very good match between the analytical results and the real situations. The contribution of this paper may be useful to deduce the closed form expressions of Bit Error Ratio (BER), the Complementary Cumulative Distribution Function (CCDF) of Peak to Average Power Ratio (PAPR), and other analytical studies which adopt the PDF.

Combustion instability due to thermo-acoustic interactions in high-speed propulsion devices such as gas turbines and rocket engines result from pressure waves with very large amplitudes propagating back and forth in the combustion chamber. Exposure to the pressure fluctuations over a long period of time can lead to a cataclysmic failure of engines. The underlying physics governing the generation of the thermo-acoustic instability is a complex interaction among heat release, turbulence, and acoustic waves. Currently, it is very difficult to accurately predict the expected level of oscillations in a combustor. Hence development of strategies and engineering solutions to mitigate thermo-acoustic instability is an active area of research in both academia and industry. In this work, we carry out numerical modeling of thermo-acoustic instability in a self-excited, laboratory scale, model rocket combustor developed at Purdue University. Two different turbulent combustion models to account for turbulence-chemistry interactions are considered in this study, the flamelet model and the transported probability density function (PDF) method. <br>In the flamelet modeling approach, detailed chemical kinetics can be easily incorporated at a relatively low cost in comparison to other turbulent combustion models and it also accounts for turbulence-chemistry interactions. The flamelet model study is divided into two parts. In first part, we examine the effect of different numerical approaches for implementing the flamelet model. In advanced modeling and simulations of turbulent combustion, the accuracy of model predictions is affected by physical model errors as well as errors that arise from the numerical implementation of models in simulation codes. Here we are mainly concerned with the effect of numerical implementation on model predictions of turbulent combustion. Particularly, we employ the flamelet/progress variable (FPV) model and examine the effect of various numerical approaches for the flamelet table integration, with presumed shapes of PDF, on the FPV modeling results. Three different presumed-PDF table integration approaches are examined in detail by employing different numerical integration strategies. The effect of the different presumed-PDF table integration approaches is examined on predictions of two real flames, a laboratory-scale turbulent free jet flame, Sandia Flame D and the self-excited resonance model rocket combustor. Significant difference is observed in the predictions both of the flames. The results in this study further support the claims made in previous studies that it is imperative to preserve the laminar flamelet structure during integration while using the flamelet model to achieve better predictions in simulations. In the second part of the flamelet modeling study, computational investigations of the coupling between the transient flame dynamics such as the ignition delay and local extinction and the thermo-acoustic instability developed in a self-excited resonance combustor to gain deep insights into the mechanisms of thermo-acoustic instability. A modeling framework that employs different flamelet models (the steady flamelet model and the flamelet/progress variable approach) is developed to enable the examination of the effect of the transient flame dynamics caused by the strong coupling of the turbulent mixing and finite-rate chemical kinetics on the occurrence of thermo-acoustic instability. The models are validated by using the available experimental data for the pressure signal. Parametric studies are performed to examine the effect of the occurrence of the transient flame dynamics, the effect of artificial amplification of the Damkohler number, and the effect of neglecting mixture fraction fluctuations on the predictions of the thermo-acoustic instability. The parametric studies reveal that the occurrence of transient flame dynamics has a strong influence on the onset of the thermo-acoustic instability. Further analysis is then conducted to localize the effect of a particular flame dynamic event, the ignition delay, on the thermo-acoustic instability. The reverse effect of the occurrence of the thermo-acoustic instability on the transient flame dynamics in the combustor is also investigated by examining the temporal evolution of the local flame events in conjunction with the pressure wave propagation. The above observed two-way coupling between the transient flame dynamics (the ignition delay) and the thermo-acoustic instability provides a plausible mechanism of the self-excited and sustained thermo-acoustic instability observed in the combustor.<br>The second turbulent combustion model considered in this study is the transported PDF method. The transported PDF method is one of the most attractive models because it treats the highly-nonlinear chemical reaction source term without a closure requirement and it is a generalized model for a wide range of turbulent combustion problems.Traditionally, the transported PDF method has been used to model low-Mach number, incompressible flows where the pressure is assumed to be thermodynamically constant. Since there is significant pressure fluctuations in the model rocket combustor, the flow is highly compressible and it is necessary to account for this compressibility in the transported PDF method. In the past there has been very little work to model compressible reactive flows using the transported PDF and no effort has been made to model thermo-acoustic instability using the transported PDF method. There is a pressing need to further examine and develop the transported PDF method for compressible reactive flows to broaden our understanding of physical phenomenon like thermo-acoustic instability, interaction between combustion and strong shock and expansion waves, coupling between acoustic and heat release which are observed in high-speed turbulent combustion problems. To address this, a modeling framework for compressible turbulent reactive flows by the using the transported PDF method is developed. This framework is validated in a series of test cases ranging from pure mixing to a supersonic turbulent jet flame. The framework is then used to study the thermo-acoustic interactions in the self-excited model rocket combustor.

We present a simple sublinear-time algorithm for sampling an arbitrary subgraph H exactly uniformly from a graph G, which the algorithm has access by performing the following types of queries: (1) uniform vertex queries, (2) degree queries, (3) neighbor queries, (4) pair queries and (5) edge sampling queries. The query complexity and running time of our algorithm are O(min{m, (m^ρ(H))/#H}) and O((m^ρ(H))/#H}), respectively, where ρ(H) is the fractional edge-cover of H and #H is the number of copies of H in G. For any clique on r vertices, i.e., H = K_r, our algorithm is almost optimal as any algorithm that samples an H from any distribution that has Ω(1) total probability mass on the set of all copies of H must perform Ω(min{m, (m^ρ(H))/(#H⋅(cr)^r)}) queries. Together with the query and time complexities of the (1±e)-approximation algorithm for the number of subgraphs H by Assadi et al. [Sepehr Assadi et al., 2018] and the lower bound by Eden and Rosenbaum [Eden and Rosenbaum, 2018] for approximately counting cliques, our results suggest that in our query model, approximately counting cliques is equivalent to exactly uniformly sampling cliques, in the sense that the query and time complexities of exactly uniform sampling and randomized approximate counting are within polylogarithmic factor of each other. This stands in interesting contrast an analogous relation between approximate counting and almost uniformly sampling for self-reducible problems in the polynomial-time regime by Jerrum, Valiant and Vazirani [Jerrum et al., 1986].

In this work, we perform an analysis, in the context of channel equalization, of two criteria that can be considered central to the field of information theoretic learning (ITL): the minimum error entropy criterion (MEEC) and the maximum correntropy criterion (MCC). An original derivation of the exact cost function of these criteria in the scenario of interest is provided and used to analyze their robustness and efficiency from a number of relevant standpoints. Another important feature of the paper is an study of the estimated versions of these cost functions, which raises several aspects regarding parameters of the canonical Parzen window estimator. The study is carried out for distinct channel and noise models, both in the combined response and parameter spaces, and also employs as benchmarks crucial metrics like the probability of bit error. The conclusions indicate under what conditions ITL criteria are particularly reliable and a number of factors that can lead to suboptimal performance.

Optimization of the spectral radius of a product for nonnegative matrices

Throughout many scientific and engineering fields, including control theory, quantum mechanics, advanced dynamics, and network theory, a great many important applications rely on the spectral decomposition of matrices. Traditional methods such as the power iteration method, Jacobi eigenvalue method, and QR decomposition are commonly used to compute the eigenvalues and eigenvectors of a square and symmetric matrix. However, these methods suffer from certain drawbacks: in particular, the power iteration method can only find the leading eigen-pair (i.e., the largest eigenvalue and its corresponding eigenvector), while the Jacobi and QR decomposition methods face significant performance limitations when facing with large scale matrices. Typically, even producing approximate eigenpairs of a general square matrix requires at least O(N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ) time complexity, where N is the number of rows of the matrix. In this work, we exploit the newly developed memristor technology to propose a low-complexity, scalable memristorbased method for deriving a set of dominant eigenvalues and eigenvectors for real symmetric non-negative matrices. The time complexity for our proposed algorithm is O(N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /Δ) (where Δ governs the accuracy). We present experimental studies to simulate the memristor-supporting algorithm, with results demonstrating that the average error for our method is within 4%, while its performance is up to 1.78X better than traditional methods.