Function Estimation Problems Research Articles

Wavelet estimation by Bayesian thresholding and model selection

Many wavelet-based algorithms have been proposed in recent years to solve the problem of function estimation from noisy samples. In particular it has been shown that threshold approaches lead to asymptotically optimal estimation and are extremely effective when dealing with real data. Working under a Bayesian perspective, in this paper we first study optimality of the hard and soft thresholding rules when the function is modelled as a stochastic process with known covariance function. Next, we consider the case where the covariance function is unknown, and propose a novel approach that models the covariance as a certain wavelet combination estimated from data by Bayesian model selection. Simulated data are used to show that the new method outperforms traditional threshold approaches as well as other wavelet-based Bayesian techniques proposed in the literature.

On the performances of a new thresholding procedure using tree structure

This paper deals with the problem of function estimation. Using the white noise model setting, we provide a method to construct a new wavelet procedure based on thresholding rules which takes advantage of the dyadic structure of the wavelet decomposition. We prove that this new procedure performs very well since, on the one hand, it is adaptive and near-minimax over a large class of Besov spaces and, on the other hand, the maximal functional space (maxiset) where this procedure attains a given rate of convergence is very large. More than this, by studying the shape of its maxiset, we prove that the new procedure outperforms the hard thresholding procedure.

Continuous Contour Monte Carlo for Marginal Density Estimation With an Application to a Spatial Statistical Model

The problem of marginal density estimation for a multivariate density function f(x) can be generally stated as a problem of density function estimation for a random vector λ(x) of dimension lower than that of x. In this article, we propose a technique, the so-called continuous Contour Monte Carlo (CCMC) algorithm, for solving this problem. CCMC can be viewed as a continuous version of the contour Monte Carlo (CMC) algorithm recently proposed in the literature. CCMC abandons the use of sample space partitioning and incorporates the techniques of kernel density estimation into its simulations. CCMC is more general than other marginal density estimation algorithms. First, it works for any density functions, even for those having a rugged or unbalanced energy landscape. Second, it works for any transformation λ(x) regardless of the availability of the analytical form of the inverse transformation. In this article, CCMC is applied to estimate the unknown normalizing constant function for a spatial autologistic model, and the estimate is then used in a Bayesian analysis for the spatial autologistic model in place of the true normalizing constant function. Numerical results on the U.S. cancer mortality data indicate that the Bayesian method can produce much more accurate estimates than the MPLE and MCMLE methods for the parameters of the spatial autologistic model.

Functional Optimal Estimation Problems and Their Solution by Nonlinear Approximation Schemes

The design of state estimators for nonlinear dynamic systems affected by disturbances is addressed in a functional optimization framework. The estimator contains an innovation function that has to be chosen within a suitably defined class of functions in such a way to minimize a cost functional given by the worst-case ratio of the ℒ p norms of the estimation error and the disturbances. Since this entails an infinite-dimensional optimization problem that under general hypotheses cannot be solved analytically, an approximate solution is sought by minimizing the cost functional over linear combinations of simple “basis functions,” represented by computational units with adjustable parameters. The selection of the parameters is made by solving a constrained nonlinear programming problem, where the constraints are given by pointwise conditions that ensure the well-definiteness of the functional and the existence of a solution. Penalty terms are introduced in the cost function to account for constraints imposed on points that result from sampling the sets to which the trajectories of the state and of the estimation error belong. To ensure an efficient covering of the sets, low-discrepancy sampling techniques are exploited that generate samples deterministically spread in a uniform way, without leaving regions of the space undersampled.

A note on Bayesian nonparametric regression function estimation

In this note the problem of nonparametric regression function estimation in a random design regression model with Gaussian errors is considered from the Bayesian perspective. It is assumed that the regression function belongs to a class of functions with a known degree of smoothness. A prior distribution on the given class can be induced by a prior on the coefficients in a series expansion of the regression function through an orthonormal system. The rate of convergence of the resulting posterior distribution is employed to provide a measure of the accuracy of the Bayesian estimation procedure defined by the posterior expected regression function. We show that the Bayes’ estimator achieves the optimal minimax rate of convergence under mean integrated squared error over the involved class of regression functions, thus being comparable to other popular frequentist regression estimators.

Application of Clustering in the Non-Parametric Estimation of Distribution Density

This paper discusses a multimodal density function estimation problem of a random vector. A comparative accuracy analysis of some popular non-parametric estimators is made by using the Monte-Carlo method. The paper demonstrates that the estimation quality increases significantly if the sample is clustered (i.e., the multimodal density function is approximated by a mixture of unimodal densities), and later on, the density estimation methods are applied separately to each cluster. In this paper, the sample is clustered using the Gaussian distribution mixture model and the EM algorithm. The highest efficiency in the analysed cases was reached by using the iterative procedure proposed by Friedman for estimating a density component corresponding to each cluster after the primary sample clustering mentioned. The Friedman procedure is based on both the projection pursuit of multivariate observations and transformation of the univariate projections into the standard Gaussian random values (using the density function estimates of these projections).

A non-parametric approach for linear system identification using principal component analysis

This paper considers the applications of principal component analysis (PCA) for signal-based linear system identification. Linear time-invariant (LTI) single-input-single-output (SISO) and multi-input-multi-output (MIMO) system frequency response function (FRF) estimation problems are formulated on the basis of the eigen-value decomposition (EVD) of the input–output measurement spectral correlation matrix. It is demonstrated that resulting algorithms for the SISO and MIMO cases are equivalent to that of the maximum likelihood (ML) and the total least squares (TLS) approaches respectively. Originating from the proposed FRF estimation scheme, a moving-segment EVD procedure is developed for SISO time-varying transfer function estimation. Based on the sensitivity of the time-domain PCA to delays/shifts between signals, an extended lagged-covariance-matrix approach is introduced for delay detection from time series.

Dictionary-based stochastic expectation-maximization for SAR amplitude probability density function estimation

In remotely sensed data analysis, a crucial problem is represented by the need to develop accurate models for the statistics of the pixel intensities. This paper deals with the problem of probability density function (pdf) estimation in the context of synthetic aperture radar (SAR) amplitude data analysis. Several theoretical and heuristic models for the pdfs of SAR data have been proposed in the literature, which have been proved to be effective for different land-cover typologies, thus making the choice of a single optimal parametric pdf a hard task, especially when dealing with heterogeneous SAR data. In this paper, an innovative estimation algorithm is described, which faces such a problem by adopting a finite mixture model for the amplitude pdf, with mixture components belonging to a given dictionary of SAR-specific pdfs. The proposed method automatically integrates the procedures of selection of the optimal model for each component, of parameter estimation, and of optimization of the number of components by combining the stochastic expectation-maximization iterative methodology with the recently developed method-of-log-cumulants for parametric pdf estimation in the case of nonnegative random variables. Experimental results on several real SAR images are reported, showing that the proposed method accurately models the statistics of SAR amplitude data.

Support vector interval regression machine for crisp input and output data

Support vector regression (SVR) has been very successful in function estimation problems for crisp data. In this paper, we propose a robust method to evaluate interval regression models for crisp input and output data combining the possibility estimation formulation integrating the property of central tendency with the principle of standard SVR. The proposed method is robust in the sense that outliers do not affect the resulting interval regression. Furthermore, the proposed method is model-free method, since we do not have to assume the underlying model function for interval nonlinear regression model with crisp input and output. In particular, this method performs better and is conceptually simpler than support vector interval regression networks (SVIRNs) which utilize two radial basis function networks to identify the upper and lower sides of data interval. Five examples are provided to show the validity and applicability of the proposed method.

Interval regression analysis using quadratic loss support vector machine

Support vector machines (SVMs) have been very successful in pattern recognition and function estimation problems for crisp data. This paper proposes a new method to evaluate interval linear and nonlinear regression models combining the possibility and necessity estimation formulation with the principle of quadratic loss SVM. This version of SVM utilizes quadratic loss function, unlike the traditional SVM. For data sets with crisp inputs and interval outputs, the possibility and necessity models have been recently utilized, which are based on quadratic programming approach giving more diverse spread coefficients than a linear programming one. The quadratic loss SVM also uses quadratic programming approach whose another advantage in interval regression analysis is to be able to integrate both the property of central tendency in least squares and the possibilistic property in fuzzy regression. However, this is not a computationally expensive way. The quadratic loss SVM allows us to perform interval nonlinear regression analysis by constructing an interval linear regression function in a high dimensional feature space. The proposed algorithm is a very attractive approach to modeling nonlinear interval data, and is model-free method in the sense that we do not have to assume the underlying model function for interval nonlinear regression model with crisp inputs and interval output. Experimental results are then presented which indicate the performance of this algorithm.

非線形写像学習のためのＰａＬＭ‐Ｔｒｅｅの提案

This paper presents a novel learning method for nonlinear mapping between arbitrary dimensional spaces. Unlike artificial neural nets, GMDH, and other methods, our method doesn't require complicated control parameters. Providing a feasible error threshold and training samples, it automatically divides the objective mapping into partially linear mappings. Since decomposed mappings are maintained by a binary tree, the linear mapping corresponding to an input is quickly selected. We call this method Partially Linear Mapping tree (PaLM-tree) . In order to estimate the most reliable linear mappings satisfying the feasible error criterion, we employ split-and-merge strategy for the decomposition. Through the experiments on function estimation, image segmentation, and camera calibration problems, we confirmed the advantages of PaLM-tree.

Fractals and Statistics: An R Package Called IFS

Iterated Function Systems (IFSs) were born in mideighties as applications of the theory of discrete dynamical systems and as useful tools to build fractals and other similar sets. Some possible applications of IFSs can be found in image processing theory in the theory of stochastic growth models and in the theory of random dynamical systems. We propose to apply (affine) IFSs to statistics. In particular, we are concerned with the problem of distribution function estimation and related quantities like the characteristic and density function.

On the quantum computational complexity of the Ising spin glass partition function and of knot invariants

It is shown that the canonical problem of classical statistical thermodynamics, the computation of the partition function, is in the case of ±J Ising spin glasses a particular instance of certain simple sums known as quadratically signed weight enumerators (QWGTs). On the other hand, it is known that quantum computing is polynomially equivalent to classical probabilistic computing with an oracle for estimating certain QWGTs. This suggests a connection between the partition function estimation problem for spin glasses and quantum computation. This connection extends to knots and graph theory via the equivalence of the Kauffman bracket polynomial and the partition function for the Potts model.

Numerical nonlinear inverse problem of determining wall heat flux

The inverse problem of determining time-variable surface heat flux in a plane wall, with constant or temperature dependent thermal properties, is numerically studied. Different kinds of incident heat flux, including rectangular waveform, are assumed. The solution is numerically solved as a function estimation problem, so that no a priori information for the functional waveforms of the unknown heat flux is needed. In all cases, a solution in the form of a piece-wise function is used to approach the incident flux. Transient temperature measurements at the boundary, from the solution of the direct problem, served as the simulated experimental data needed as input for the inverse analysis. Both direct and inverse heat conduction problems are solved using the network simulation method. The solution is obtained step-by-step by minimising the classical functional that compares the above input data with those obtained from the solution of the inverse problem. A straight line of variable slope and length is used for each one of the stretches of the desired solution. The influence of random error, number of functional terms and the effect of sensor location are studied. In all cases, the results closely agree with the solution.

Nonparametric M-regression with free knot splines

Many problems of practical interest can be formulated as the nonparametric estimation of a certain function such as a regression function, logistic or other generalized regression function, density function, conditional density function, hazard function, or conditional hazard function. Extended linear modeling provides a convenient theoretical framework for using polynomial splines and their selected tensor products in such function estimation problems and especially for obtaining rates of convergence of the resulting estimates in a unified manner. For a long time the theoretical results were restricted to fixed knot splines and to log-likelihood functions that were twice continuously differentiable. Recently, Stone and Huang extended the theory to handle free knot splines. In the present paper, the theory is further extended to handle contexts in which the log-likelihood function may not be differentiable. Specifically, we establish rates of convergence for estimation based on free knot splines in the context of nonparametric regression corresponding to M-estimates, which includes least absolute deviations (LAD) regression, quantile regression, and robust regression as special cases.

Volterra integral equations solved in Fredholm form using Walsh functions

Recently Walsh function methods have been developed for the numerical solution of several classes of problems, mainly linear and nonlinear integral equations of both Volterra and Fredholm types. In addition, modifications of the basic approach have led to the solution of functional differential equations, variational problems and parameter estimation problems. Linear Volterra integral equations are re-written as Fredholm integral equations with appropriately modified kernels. In this Fredholm equation form, the Walsh function solution method is more efficient than directly solving the Volterra equation. Walsh function methods are spectral methods but they have a natural grid interpretation. Multigrid methods and a variation on the use of Richardson extrapolation are used on six well known Volterra test problems, re-written in Fredholm form, to illustrate that these methods provide effective and efficient solution methods.

Backfitting neural networks

Regression and classification problems can be viewed as special cases of the problem of function estimation. It is rather well known that a two-layer perceptron with sigmoidal transformation functions can approximate any continuous function on the compact subsets of RP if there are sufficient number of hidden nodes. In this paper, we present an algorithm for fitting perceptron models, which is quite different from the usual backpropagation or Levenberg-Marquardt algorithm. This new algorithm based on backfitting ensures a better convergence than backpropagation. We have also used resampling techniques to select an ideal number of hidden nodes automatically using the training data itself. This resampling technique helps to avoid the problem of overfitting that one faces for the usual perceptron learning algorithms without any model selection scheme. Case studies and simulation results are presented to illustrate the performance of this proposed algorithm.

Inverse determination of the specific heat of foods

Simulation of heat conduction processes in foods requires a knowledge of the temperature dependence of their thermal properties, conductivity ( k) and specific heat ( c e). In this work a new procedure is proposed for the inverse estimation of c e in a 1-D solid under convective and adiabatic boundary conditions, in which conductivity is constant or temperature-dependent. The inverse problem is treated as a function estimation problem so that no prior information is needed about the form of the dependence. The estimation is obtained by means of a piece-wise function, whose stretches (both slope and size) is continuously adjusted by an iterative routine that minimizes the classical functional for this kind of problems. This routine is combined with the Network simulation method as the numerical technique. The input data of this inverse problem are obtained from the solution of the direct problem, at a particular location in the food, modified by a normal distribution random error. Applications to real products are realized proving that it is possible to reach precise estimations of c e with the proposed method.

An introduction to reproducing kernel hilbert spaces and why they are so useful

We review some of the basic facts about reproducing kernel Hilbert spaces (RKHS), and the solution of optimization problems in RKHS, These facts provide some clues to how useful RKHS-based methods can be in curve fitting, function estimation, model description, model fitting and ill-posed inverse problems. A number of references are made to mostly older works of the author, colleagues and former students - not an attempt at a balanced review of the literature. The growth of the further development and application of RKHS based methods is demonstrated in the other papers in the invited Session INV-2 of this 13th IFAC Symposium on System Identification (SYSID-2003), and elsewhere in SISID-2003. A recent advanced google search for "reproducing kernel" turned up over 4400 entries.

Nonparametric estimation under long memory dependence

Modelling dependence structure by means of long memory processes appears of great interest for many applications. This paper aims to show how nonparametric kernel estimators behave asymptotically under this dependence structure. We will consider a wide class of functional estimation problems, including distribution function, hazard function and density function, and we will give rates of weak consistency and of mean squared consistency. A main feature of this paper is to deal with a general form of long dependence processes, without assuming any Gaussian shape for the data or any parametric form for the covariances sequence. Finally, we will show how these results apply naturally to many classical parametric processes, and special attention will be paid to the Fractional Brownian Motion and to FARIMA processes.