Function Estimation Problems Research Articles

Online estimation of objective function for continuous-time deterministic systems

We developed two online data-driven methods for estimating an objective function in continuous-time linear and nonlinear deterministic systems. The primary focus addressed the challenge posed by unknown input dynamics (control mapping function) in the expert system, a critical element for an online solution of the problem. Our methods leverage both the learner’s and expert’s data for effective problem-solving. The first approach, which is model-free, estimates the expert’s policy and integrates it into the learner agent to approximate the objective function associated with the optimal policy. The second approach estimates the input dynamics from the learner’s data and combines it with the expert’s input-state observations to tackle the objective function estimation problem. Compared to other methods for deterministic systems that rely on both the learner’s and expert’s data, our approaches offer reduced complexity by eliminating the need to estimate an optimal policy after each objective function update. We conduct a convergence analysis of the estimation techniques using Lyapunov-based methods. Numerical experiments validate the effectiveness of our developed methods.

An Online Database of Benchmark Problems for Verification of Inverse Problems Computer Codes

Verification and validation are a fundamental step in the development process of computer codes, particularly when seeking to numerically evaluate physical phenomena. Multiple methodologies can be applied, one of which is the use of benchmark solutions. The field of inverse problems is no exception, for not only the forward problem must be verified and validated, but also the inversion technique (such as minimization methods, Bayesian inference, and level-set methods). To address the latter, this work outlines an online database of benchmark inverse problems, for the purposes of verifying computer codes for inverse problems. This online database provides test problems with suggested values for the exact parameters and regularization parameters, as well as synthetic data and their respective estimated values. Both parameter and function estimation problems are available. This database will be continuously updated over time, with new problems and/or solutions via different techniques. Users are also free to contribute with problems/solutions of their own. This data being freely and easily accessible online will be of great value for young and upcoming researchers in the field of inverse problems. The database can be accessed at http://inversetoolkit.uff.br/.

Penalized Whittle likelihood for spatial data

Inference for spatial data is challenging because fitting an appropriate parametric model is often difficult. The penalized likelihood-type approach has been successfully developed for various nonparametric function estimation problems in time series analysis. However, it has not been well developed in spatial analysis. In this paper, a penalized Whittle likelihood approach is developed for nonparametric estimation of spectral density functions for regularly spaced spatial data. In particular, the estimated spectral density is the minimizer of a criterion which is developed based on the Whittle likelihood and a penalty for roughness. This approach aggregates several popular nonparametric density estimation methods into a coherent framework. Asymptotic properties of the proposed estimator are derived under mild assumptions without assuming Gaussianity. In addition, a computationally efficient method is developed to optimize the penalized likelihood function. Simulation results and real data examples are also provided to illustrate the finite sample performances of the methodology.

Minimax estimation in multi-task regression under low-rank structures

This study investigates the minimaxity of a multi-task nonparametric regression problem. We formulate a simultaneous function estimation problem based on information pooling across multiple experiments under a low-dimensional structure. A nonparametric reduced rank regression estimator based on the nuclear norm penalisation scheme is proposed to incorporate the low-dimensional structure in the estimation process. A rank of a set of functions is defined in terms of their Fourier coefficients to formally characterise the dependence structure among functions. Minimax upper and lower bounds are established under various asymptotic scenarios to examine the role of the low-rank structure in determining optimal rates of convergence. The results confirm that exploiting the low-rank structure can significantly improve the convergence rate for the simultaneous estimation of multiple functions. The results also imply that the proposed estimator is rate optimal in the minimax sense for the rank-constraint Sobolev class of vector-valued functions.

Viscosity and effusion rate identification from free surface data

Based on new explicit expressions for sensitivity coefficients, this study proposes a new method to solve two inverse problems in time-dependent three-dimensional gravity-driven thin film flow down an inclined plane governed by free-surface lubrication approximation. The flow is developed by time-dependent injection through a circular hole in the plane. The first inverse problem, a parameter estimation problem, consists in simultaneously estimating the temperature-dependent viscosity and a constant injection. The second inverse problem, a function estimation problem, involves the determination of the time-dependent injection. It is shown here that using the derived sensitivity coefficients, a parameter estimation approach can still be used for the second inverse problem, efficiently and accurately, and without the need to solve the adjoint and sensitivity problems as well as additional direct problem solutions.

Regularization of Fourier Series with Approximate Coefficients for the Problem of Phase Probability Density Function Estimation

Regularization of Fourier Series with Approximate Coefficients for the Problem of Phase Probability Density Function Estimation

Metropolis–Hastings via Classification

This article develops a Bayesian computational platform at the interface between posterior sampling and optimization in models whose marginal likelihoods are difficult to evaluate. Inspired by contrastive learning and Generative Adversarial Networks (GAN), we reframe the likelihood function estimation problem as a classification problem. Pitting a Generator, who simulates fake data, against a Classifier, who tries to distinguish them from the real data, one obtains likelihood (ratio) estimators which can be plugged into the Metropolis–Hastings algorithm. The resulting Markov chains generate, at a steady state, samples from an approximate posterior whose asymptotic properties we characterize. Drawing upon connections with empirical Bayes and Bayesian misspecification, we quantify the convergence rate in terms of the contraction speed of the actual posterior and the convergence rate of the Classifier. Asymptotic normality results are also provided which justify the inferential potential of our approach. We illustrate the usefulness of our approach on examples which have challenged for existing Bayesian likelihood-free approaches. Supplementary materials for this article are available online.

Estimation of Probability Density Function in the Case of Multiplicative Noise

In the present paper, we consider the problem of probability density function estimation. Our data has multiplicative noise; therefore, we cannot use direct methods. Our method is based on estimation of coefficients of the Fourier transform for the density function.

Regularization of Fourier series with approximate coefficients for the problem of phase probability density function estimation

Regularization of Fourier series with approximate coefficients for the problem of phase probability density function estimation

Bernstein polynomial distribution estimators and the Dvoretzky–Kiefer–Wolfowitz inequality

The problem of nonparametric distribution function estimation using the Bernstein polynomials is investigated. It is proved that the Bernstein polynomial estimator that was most often examined so far does not satisfy the Dvoretzky–Kiefer–Wolfowitz inequal

Estimation of Functional Form of Time-Dependent Heat Transfer Coefficient Using an Accurate and Robust Parameter Estimation Approach: An Inverse Analysis

This paper presents a numerical method to address function estimation problems in inverse heat transfer problems using parameter estimation approach without prior information on the functional form of the variable to be estimated. Using an inverse analysis, the functional form of a time-dependent heat transfer coefficient is estimated efficiently and accurately. The functional form of the heat transfer coefficient is assumed unknown and the inverse heat transfer problem should be treated using a function estimation approach by solving sensitivity and adjoint problems during the minimization process. Based on proposing a new sensitivity matrix, however, the functional form can be estimated in an accurate and very efficient manner using a parameter estimation approach without the need for solving the sensitivity and adjoint problems and imposing extra computational cost, mathematical complexity, and implementation efforts. In the proposed sensitivity analysis scheme, all sensitivity coefficients can be computed in only one direct problem solution at each iteration. In this inverse heat transfer problem, the body shape is irregular and meshed using a body-fitted grid generation method. The direct heat conduction problem is solved using the finite-difference method. The steepest-descent method is used as a minimization algorithm to minimize the defined objective function and the termination of the minimization process is carried out based on the discrepancy principle. A test case with three different functional forms and two different measurement errors is considered to show the accuracy and efficiency of the used inverse analysis.

Novel pricing strategies for revenue maximization and demand learning using an exploration-exploitation framework.

The price demand relation is a fundamental concept that models how price affects the sale of a product. It is critical to have an accurate estimate of its parameters, as it will impact the company’s revenue. The learning has to be performed very efficiently using a small window of a few test points, because of the rapid changes in price demand parameters due to seasonality and fluctuations. However, there are conflicting goals when seeking the two objectives of revenue maximization and demand learning, known as the learn/earn trade-off. This is akin to the exploration/exploitation trade-off that we encounter in machine learning and optimization algorithms. In this paper, we consider the problem of price demand function estimation, taking into account its exploration–exploitation characteristic. We design a new objective function that combines both aspects. This objective function is essentially the revenue minus a term that measures the error in parameter estimates. Recursive algorithms that optimize this objective function are derived. The proposed method outperforms other existing approaches.

Nonparametric local linear estimation of the relative error regression function for twice censored data

This paper deals with the problem of nonparametric relative error regression function estimation for twice censored data. A new estimate is proposed, which is built by combining the local linear approach and relative error estimation. A uniform almost sure consistency with rate over a compact set is established. A numerical study is carried out to assess the performance of the proposed estimator. Practical results indicate the robustness of the new estimate compared to other existing estimators in the presence of censored and outliers datum.

Fast multivariate empirical cumulative distribution function with connection to kernel density estimation

The problem of computing empirical cumulative distribution functions (ECDF) efficiently on large, multivariate datasets, is revisited. Computing an ECDF at one evaluation point requires O(N) operations on a dataset composed of N data points. Therefore, a direct evaluation of ECDFs at N evaluation points requires a quadratic O(N2) operations, which is prohibitive for large-scale problems. Two fast and exact methods are proposed and compared. The first one is based on fast summation in lexicographical order, with a O(Nlog⁡N) complexity and requires the evaluation points to lie on a regular grid. The second one is based on the divide-and-conquer principle, with a O(Nlog⁡(N)(d−1)∨1) complexity and requires the evaluation points to coincide with the input points. The two fast algorithms are described and detailed in the general d-dimensional case, and numerical experiments validate their speed and accuracy. Secondly, a direct connection between cumulative distribution functions and kernel density estimation (KDE) is established for a large class of kernels. This connection paves the way for fast exact algorithms for multivariate kernel density estimation and kernel regression. Numerical tests with the Laplacian kernel validate the speed and accuracy of the proposed algorithms. A broad range of large-scale multivariate density estimation, cumulative distribution estimation, survival function estimation and regression problems can benefit from the proposed numerical methods.

Computing partition functions in the one-clean-qubit model

We present a method to approximate partition functions of quantum systems using mixed-state quantum computation. For positive-semidefinite Hamiltonians, our method has an expected running-time that is almost linear in ${[M/({\ensuremath{\epsilon}}_{\mathrm{rel}}\mathcal{Z})]}^{2}$, where $M$ is the dimension of the quantum system, $\mathcal{Z}$ is the partition function, and ${\ensuremath{\epsilon}}_{\mathrm{rel}}$ is the relative precision. It is based on approximations of the exponential operator as linear combinations of certain operators related to block-encoding of Hamiltonians or Hamiltonian evolutions. The trace of each operator is estimated using a standard algorithm in the one-clean-qubit model. For large values of $\mathcal{Z}$, our method may run faster than exact classical methods, whose complexities are polynomial in $M$. We also prove that a version of the partition function estimation problem within additive error is complete for the so-called DQC1 complexity class, suggesting that our method provides a superpolynomial speedup for certain parameter values. To attain a desired relative precision, we develop a classical procedure based on a sequence of approximations within predetermined additive errors that may be of independent interest.

Supplement to 'Estimation in Functional Regression for General Exponential Families'

In this supplemental article, we introduce some useful results in spectral theory and perturbation theory. Some of the results are well-established. We briefly review them for the purpose of easy reference. We derive a novel bound for the perturbation of eigenprojections, which plays a key role in the slope function estimation problem.

Function Estimation in Inverse Heat Transfer Problems Based on Parameter Estimation Approach

A new sensitivity analysis scheme is presented based on explicit expressions for sensitivity coefficients to estimate timewise varying heat flux in heat conduction problems over irregular geometries using the transient readings of a single sensor. There is no prior information available on the functional form of the unknown heat flux; hence, the inverse problem is regarded as a function estimation problem and sensitivity and adjoint problems are involved in the solution of the inverse problem to recover the unknown heat flux. However, using the proposed sensitivity analysis scheme, one can compute all sensitivity coefficients explicitly in only one direct problem solution at each iteration without the need for solving the sensitivity and adjoint problems. In other words, the functional form of the unknown heat flux can be numerically estimated by using the parameter estimation approach. In this method, the irregular shape of heat-conducting body is meshed using the boundary-fitted grid generation (elliptic) method. Explicit expressions are given to compute the sensitivity coefficients efficiently and the steepest-descent method is used as the minimization method to minimize the objective function and reach the solution. Three test cases are presented to confirm the accuracy and efficiency of the proposed inverse analysis.

Semisupervised Learning on Graphs With an Alternating Diffusion Process.

Graph-based semisupervised learning is of great importance in many effective learning systems, particularly in agnostic settings where no parametric information or other prior knowledge about the data distribution is available. It leverages the graph structure to propagate labels from labeled nodes to unlabeled ones. Two separate stages are usually involved: constructing an affinity graph and propagating labels on the graph for transductive inference. It is suboptimal to manage them independently, as the correlation between the affinity graph and the labels would not be fully exploited. In this article, we integrate these two stages into one unified framework by formulating the graph construction as a regularized function estimation problem, similar to label propagation. We then propose an alternating diffusion process to solve them alternately, which allows us to learn the graph and unknown labels in an iterative fashion. With the proposed framework, we can construct a dynamic graph adapted to the given and predicted labels iteratively, resulting in more accurate and robust label propagation performance. Extensive experiments on synthetic data and various real-world data have demonstrated the superiority of the proposed method compared with other state-of-the-art methods.

Influence of adaptive thresholding on peaks detection in audio data

Many audio analysis systems employ peak picking procedure to produce the final decision. A typical scheme uses a thresholding function to minimise detection errors where its form depends on the structure of the input signal. The paper covers the problem of an adaptive thresholding function estimation. Using the genetic algorithm to optimise the components of the thresholding function we have determined the level of importance of individual local statistics on the final function representation. The proposed method has been used to tune the peak detection procedure to identify the change points in an audio signal. In the result of the heuristic configuration, the best accuracy of segment boundaries have been obtained for thresholding function built on top of two local statistics of the detection function and constant value. Finally, as an example, a comparison with the state–of–the–art scheme for audio segmentation was performed.

Sparse RKHS estimation via globally convex optimization and its application in LPV-IO identification

Function estimation using the Reproducing Kernel Hilbert Space (RKHS) framework is a powerful tool for identification of a general class of nonlinear dynamical systems without requiring much a priori information on model orders and nonlinearities involved. However, the high degrees-of-freedom (DOFs) of RKHS estimators has its price, as in case of large scale function estimation problems, they often require a serious amount of data samples to explore the search space adequately for providing high-performance model estimates. In cases where nonlinear dynamic relations can be expressed as a sum of functions, the literature proposes solutions to this issue by enforcing sparsity for adequate restriction of the DOFs of the estimator, resulting in parsimonious model estimates. Unfortunately, all existing solutions are based on greedy approaches, leading to optimization schemes which cannot guarantee convergence to the global optimum. In this paper, we propose an ℓ1-regularized non-parametric RKHS estimator which is the solution of a quadratic optimization problem. Effectiveness of the scheme is demonstrated on the non-parametric identification problem of LPV-IO models where the method solves simultaneously (i) the model order selection problem (in terms of number of input–output lags and input delay in the model structure) and (ii) determining the unknown functional dependency of the model coefficients on the scheduling variable directly from data. The paper also provides an extensive simulation study to illustrate effectiveness of the proposed scheme.