Log-concave Density Research Articles

Faster High-accuracy Log-concave Sampling via Algorithmic Warm Starts

It is a fundamental problem to understand the complexity of high-accuracy sampling from a strongly log-concave density π on ℝ d . Indeed, in practice, high-accuracy samplers such as the Metropolis-adjusted Langevin algorithm (MALA) remain the de facto gold standard; and in theory, via the proximal sampler reduction, it is understood that such samplers are key for sampling even beyond log-concavity (in particular, for sampling under isoperimetric assumptions). This article improves the dimension dependence of this sampling problem to \(\widetilde{O}(d^{1/2})\) . The previous best result for MALA was \(\widetilde{O}(d)\) . This closes the long line of work on the complexity of MALA and, moreover, leads to state-of-the-art guarantees for high-accuracy sampling under strong log-concavity and beyond (thanks to the aforementioned reduction). Our starting point is that the complexity of MALA improves to \(\widetilde{O}(d^{1/2})\) , but only under a warm start (an initialization with constant Rényi divergence w.r.t. π). Previous algorithms for finding a warm start took O(d) time and thus dominated the computational effort of sampling. Our main technical contribution resolves this gap by establishing the first \(\widetilde{O}(d^{1/2})\) Rényi mixing rates for the discretized underdamped Langevin diffusion. For this, we develop new differential-privacy-inspired techniques based on Rényi divergences with Orlicz–Wasserstein shifts, which allow us to sidestep longstanding challenges for proving fast convergence of hypocoercive differential equations.

A new computational framework for log-concave density estimation

In statistics, log-concave density estimation is a central problem within the field of nonparametric inference under shape constraints. Despite great progress in recent years on the statistical theory of the canonical estimator, namely the log-concave maximum likelihood estimator, adoption of this method has been hampered by the complexities of the non-smooth convex optimization problem that underpins its computation. We provide enhanced understanding of the structural properties of this optimization problem, which motivates the proposal of new algorithms, based on both randomized and Nesterov smoothing, combined with an appropriate integral discretization of increasing accuracy. We prove that these methods enjoy, both with high probability and in expectation, a convergence rate of order 1/T up to logarithmic factors on the objective function scale, where T denotes the number of iterations. The benefits of our new computational framework are demonstrated on both synthetic and real data, and our implementation is available in a github repository LogConcComp (Log-Concave Computation).

Maximum likelihood estimation of log-concave densities on tree space

Phylogenetic trees are key data objects in biology, and the method of phylogenetic reconstruction has been highly developed. The space of phylogenetic trees is a nonpositively curved metric space. Recently, statistical methods to analyze samples of trees on this space are being developed utilizing this property. Meanwhile, in Euclidean space, the log-concave maximum likelihood method has emerged as a new nonparametric method for probability density estimation. In this paper, we derive a sufficient condition for the existence and uniqueness of the log-concave maximum likelihood estimator on tree space. We also propose an estimation algorithm for one and two dimensions. Since various factors affect the inferred trees, it is difficult to specify the distribution of a sample of trees. The class of log-concave densities is nonparametric, and yet the estimation can be conducted by the maximum likelihood method without selecting hyperparameters. We compare the estimation performance with a previously developed kernel density estimator numerically. In our examples where the true density is log-concave, we demonstrate that our estimator has a smaller integrated squared error when the sample size is large. We also conduct numerical experiments of clustering using the Expectation-Maximization algorithm and compare the results with k-means++ clustering using Fréchet mean.

An example of a non-log-concave distribution where the difference has a log-concave density

An example of a non-log-concave distribution where the difference has a log-concave density

Solution to the n-bubble problem on mathbb {R}^1 with log-concave density

We study the n-bubble problem on mathbb {R}^1 with a prescribed density function f that is even, radially increasing, and satisfies a log-concavity requirement. Under these conditions, we find that isoperimetric solutions can be identified for an arbitrary number of regions, and that these solutions have a well-understood and regular structure. This generalizes recent work done on the density function |x |^p and stands in contrast to log-convex density functions which are known to have no such regular structure.

Exact solutions in log-concave maximum likelihood estimation

We study probability density functions that are log-concave. Despite the space of all such densities being infinite-dimensional, the maximum likelihood estimate is the exponential of a piecewise linear function determined by finitely many quantities, namely the function values, or heights, at the data points. We explore in what sense exact solutions to this problem are possible. First, we show that the heights given by the maximum likelihood estimate are generically transcendental. For a cell in one dimension, the maximum likelihood estimator is expressed in closed form using the generalized W-Lambert function. Even more, we show that finding the log-concave maximum likelihood estimate is equivalent to solving a collection of polynomial-exponential systems of a special form. Even in the case of two equations, very little is known about solutions to these systems. As an alternative, we use Smale's α-theory to refine approximate numerical solutions and to certify solutions to log-concave density estimation.

Transport-majorization to analytic and geometric inequalities

We introduce a transport-majorization argument that establishes a majorization in the convex order between two densities, based on control of the gradient of a transportation map between them. As applications, we give elementary derivations of some delicate Fourier analytic inequalities, which in turn yield geometric “slicing-inequalities” in both continuous and discrete settings. As a further consequence of our investigation we prove that any strongly log-concave probability density majorizes the Gaussian density and thus the Gaussian density maximizes the Rényi and Tsallis entropies of all orders among all strongly log-concave densities.

Spectral gap of replica exchange Langevin diffusion on mixture distributions

Langevin diffusion (LD) is one of the main workhorses for sampling problems. However, its convergence rate can be significantly reduced if the target distribution is a mixture of multiple densities, especially when each component density concentrates around a different mode. Replica exchange Langevin diffusion (ReLD) is a sampling method that can circumvent this issue. This approach can be further extended to multiple replica exchange Langevin diffusion (mReLD). While ReLD and mReLD have been used extensively in statistics, molecular dynamics, and other applications, there is limited existing analysis on its convergence rate and choices of the temperatures. This paper addresses these problems assuming the target distribution is a mixture of log-concave densities. We show ReLD can obtain constant or better convergence rates. We also show mReLD with K additional LDs can achieve the same results while the exchange frequency only needs to be (1/K)-th power of the one in ReLD.

Confidence bands for a log-concave density

We present a new approach for inference about a univariate log-concave distribution: Instead of using the method of maximum likelihood, we propose to incorporate the log-concavity constraint in an appropriate nonparametric confidence set for the cdf F. This approach has the advantage that it automatically provides a measure of statistical uncertainty and it thus overcomes a marked limitation of the maximum likelihood estimate. In particular, we show how to construct confidence bands for the density that have a finite sample guaranteed confidence level. The nonparametric confidence set for F which we introduce here has attractive computational and statistical properties: It allows to bring modern tools from optimization to bear on this problem via difference of convex programming, and it results in optimal statistical inference. We show that the width of the resulting confidence bands converges at nearly the parametric rate when the log density is k-affine.

Oracle lower bounds for stochastic gradient sampling algorithms

We consider the problem of sampling from a strongly log-concave density in Rd, and prove an information theoretic lower bound on the number of stochastic gradient queries of the log density needed. Several popular sampling algorithms (including many Markov chain Monte Carlo methods) operate by using stochastic gradients of the log density to generate a sample; our results establish an information theoretic limit for all these algorithms. We show that for every algorithm, there exists a well-conditioned strongly log-concave target density for which the distribution of points generated by the algorithm would be at least ε away from the target in total variation distance if the number of gradient queries is less than Ω(σ2d/ε2), where σ2d is the variance of the stochastic gradient. Our lower bound follows by combining the ideas of Le Cam deficiency routinely used in the comparison of statistical experiments along with standard information theoretic tools used in lower bounding Bayes risk functions. To the best of our knowledge our results provide the first nontrivial dimension-dependent lower bound for this problem.

Inference for Local Parameters in Convexity Constrained Models

ABSTRACT In this article, we develop automated inference methods for “local” parameters in a collection of convexity constrained models based on the natural constrained tuning-free estimators. A canonical example is given by the univariate convex regression model, in which automated inference is drawn for the function value, the function derivative at a fixed interior point, and the anti-mode of the convex regression function, based on the widely used tuning-free, piecewise linear convex least squares estimator (LSE). The key to our inference proposal in this model is a pivotal joint limit distribution theory for the LS estimates of the local parameters, normalized appropriately by the length of certain data-driven linear piece of the convex LSE. Such a pivotal limiting distribution instantly gives rise to confidence intervals for these local parameters, whose construction requires almost no more effort than computing the convex LSE itself. This inference method in the convex regression model is a special case of a general inference machinery that covers a number of convexity constrained models in which a limit distribution theory is available for model-specific estimators. Concrete models include: (i) log-concave density estimation, (ii) s-concave density estimation, (iii) convex nonincreasing density estimation, (iv) concave bathtub-shaped hazard function estimation, and (v) concave distribution function estimation from corrupted data. The proposed confidence intervals for all these models are proved to have asymptotically exact coverage and oracle length, and require no further information than the estimator itself. We provide extensive simulation evidence that validates our theoretical results. Real data applications and comparisons with competing methods are given to illustrate the usefulness of our inference proposals. Supplementary materials for this article are available online.

Random concave functions

Spaces of convex and concave functions appear naturally in theory and applications. For example, convex regression and log-concave density estimation are important topics in nonparametric statistics. In stochastic portfolio theory, concave functions on the unit simplex measure the concentration of capital, and their gradient maps define novel investment strategies. The gradient maps may also be regarded as optimal transport maps on the simplex. In this paper we construct and study probability measures supported on spaces of concave functions. These measures may serve as prior distributions in Bayesian statistics and Cover’s universal portfolio, and induce distribution-valued random variables via optimal transport. The random concave functions are constructed on the unit simplex by taking a suitably scaled (mollified, or soft) minimum of random hyperplanes. Depending on the regime of the parameters, we show that as the number of hyperplanes tends to infinity there are several possible limiting behaviors. In particular, there is a transition from a deterministic almost sure limit to a nontrivial limiting distribution that can be characterized using convex duality and Poisson point processes.

Local continuity of log-concave projection, with applications to estimation under model misspecification

The log-concave projection is an operator that maps a d-dimensional distribution P to an approximating log-concave density. It is known that, with suitable metrics on the underlying spaces, this projection is continuous, but not uniformly continuous. In this work, we prove a local uniform continuity result for log-concave projection – in particular, establishing that this map is locally Hölder-(1/4) continuous. A matching lower bound verifies that this exponent cannot be improved. We also examine the implications of this continuity result for the empirical setting – given a sample drawn from a distribution P, we bound the squared Hellinger distance between the log-concave projection of the empirical distribution of the sample, and the log-concave projection of P. In particular, this yields interesting statistical results for the misspecified setting, where P is not itself log-concave.

High-dimensional central limit theorems by Stein’s method

We obtain explicit error bounds for the d-dimensional normal approximation on hyperrectangles for a random vector that has a Stein kernel, or admits an exchangeable pair coupling, or is a nonlinear statistic of independent random variables or a sum of n locally dependent random vectors. We assume the approximating normal distribution has a nonsingular covariance matrix. The error bounds vanish even when the dimension d is much larger than the sample size n. We prove our main results using the approach of Götze (1991) in Stein’s method, together with modifications of an estimate of Anderson, Hall and Titterington (1998) and a smoothing inequality of Bhattacharya and Rao (1976). For sums of n independent and identically distributed isotropic random vectors having a log-concave density, we obtain an error bound that is optimal up to a logn factor. We also discuss an application to multiple Wiener–Itô integrals.

Active set algorithms for estimating shape-constrained density ratios

In many instances, imposing a constraint on the shape of a density is a reasonable and flexible assumption. It offers an alternative to parametric models, which can be too rigid, and to other nonparametric methods, which require the choice of tuning parameters. The nonparametric estimation of log-concave or log-convex density ratios is treated by means of active set algorithms in a unified framework. In the setting of log-concave densities, the new algorithm is similar to, but substantially faster than, previously considered active set methods. Log-convexity, on the other hand, is a less common shape-constraint, described by some authors as “tail inflation”. The active set method proposed here is novel in this context. As a by-product, new goodness-of-fit tests of single hypotheses are formulated and are shown to be more powerful than higher criticism tests in a simulation study.

Optimal dual quantizers of [formula omitted][formula omitted]-concave distributions: Uniqueness and Lloyd like algorithm

We establish for dual quantization the counterpart of Kieffer’s uniqueness result for compactly supported one dimensional probability distributions having a log-concave density (also called strongly unimodal): for such distributions, Lr-optimal dual quantizers are unique at each level N, the optimal grid being the unique critical point of the quantization error. An example of non-strongly unimodal distribution for which uniqueness of critical points fails is exhibited. In the quadratic r=2 case, we propose an algorithm to compute the unique optimal dual quantizer. It provides a counterpart of Lloyd’s method I algorithm in a Voronoi framework (see [14] and [15]). Finally semi-closed forms of Lr-optimal dual quantizers are established for power distributions on compacts intervals and truncated exponential distributions.

High-dimensional nonparametric density estimation via symmetry and shape constraints

We tackle the problem of high-dimensional nonparametric density estimation by taking the class of log-concave densities on Rp and incorporating within it symmetry assumptions, which facilitate scalable estimation algorithms and can mitigate the curse of dimensionality. Our main symmetry assumption is that the super-level sets of the density are K-homothetic (i.e., scalar multiples of a convex body K⊆Rp). When K is known, we prove that the K-homothetic log-concave maximum likelihood estimator based on n independent observations from such a density achieves the minimax optimal rate of convergence with respect to, for example, squared Hellinger loss, of order n−4/5, independent of p. Moreover, we show that the estimator is adaptive in the sense that if the data generating density admits a special form, then a nearly parametric rate may be attained. We also provide worst case and adaptive risk bounds in cases where K is only known up to a positive definite transformation, and where it is completely unknown and must be estimated nonparametrically. Our estimation algorithms are fast even when n and p are on the order of hundreds of thousands, and we illustrate the strong finite-sample performance of our methods on simulated data.

Farklı Dağılımlar ve Farklı Çekirdek Fonksiyonları için ROC Eğrisi Tahmin Yöntemlerinin Performanslarının İncelenmesi

Objective: Receiver operating characteristic (ROC) curve is a statistical method used to examine the actual effectiveness of a diagnostic test or a biomarker in a comprehensive and reliable way. Several methods have been proposed to estimate ROC curve properly. The aim of the present study is to compare recent ROC curve estimation methods for different distribution and sample sizes. Material and Methods: Log-concave density and smooth log-concave density estimate based ROC curve estimation, kernel based ROC curve estimation with Gaussian, Epanechnikov, rectangular, triangular kernels, and binormal ROC estimation methods were compared for different simulation scenarios. Results: The ROC curve estimation methods based on kernel estimates gave their best performances when the biomarker values of non-diseased group are normal but the biomarker values of the diseased group are right-skewed, with a notable difference from other methods. Epanechnikov and rectangular kernel methods yielded better performance than other kernel methods in small sample sizes; but this difference disappeared as the sample size increased. The methods based on kernel or log-concave density estimate gave their worst results for the simulation scenario where the data were nonnormal but symmetric. Conclusion: The performances of the other methods examined in the study exceeded the performance of the binormal method in highly skewed data in both groups and when the distribution of diseased and non-diseased populations were right-skewed and normal, respectively.

Sharp Moment-Entropy Inequalities and Capacity Bounds for Symmetric Log-Concave Distributions

We show that the uniform distribution minimizes entropy among all one-dimensional symmetric log-concave distributions with fixed variance, as well as various generalizations of this fact to R\'enyi entropies of orders less than 1 and with moment constraints involving $p$-th absolute moments with $p\leq 2$. As consequences, we give new capacity bounds for additive noise channels with symmetric log-concave noises, as well as for timing channels involving positive signal and noise where the noise has a decreasing log-concave density. In particular, we show that the capacity of an additive noise channel with symmetric, log-concave noise under an average power constraint is at most 0.254 bits per channel use greater than the capacity of an additive Gaussian noise channel with the same noise power. Consequences for reverse entropy power inequalities and connections to the slicing problem in convex geometry are also discussed.

Modal linear regression using log-concave distributions

The modal linear regression suggested by Yao and Li (Scand J Stat 41(3):656–671, 2014) models the conditional mode of a response Y given a vector of covariates $$\mathbf{z }$$ as a linear function of $$\mathbf{z }$$ . To identify the conditional mode of Y given $$\mathbf{z }$$ , existing methods utilize a kernel density estimator to obtain the distribution of Y given $$\mathbf{z }$$ . Like other kernel-based methods, these require a suitable choice of tuning parameters, and no unified objective function exists for estimating regression parameters. In this paper, we propose a model-based modal linear regression using a family of log-concave distributions. The proposed method does not require tuning parameters and enables us to construct an explicit likelihood function. To estimate the regression parameters with an estimated log-concave density, we turned the log-likelihood into a sum of affine functions using a dual representation of piecewise linear concave functions so that well-known linear programming techniques can be adopted. Simulation studies reveal that the proposed method produces more efficient estimators than kernel-based methods. A real data example is also presented to illustrate the applicability of the proposed method.