On discriminating between lognormal and Pareto tail: an unsupervised mixture-based approach

Marco Bee

doi:10.1007/s11634-022-00497-4

Abstract

Many stochastic models in economics and finance are described by distributions with a lognormal body. Testing for a possible Pareto tail and estimating the parameters of the Pareto distribution in these models is an important topic. Although the problem has been extensively studied in the literature, most applications are characterized by some weaknesses. We propose a method that exploits all the available information by taking into account the data generating process of the whole population. After estimating a lognormal–Pareto mixture with a known threshold via the EM algorithm, we exploit this result to develop an unsupervised tail estimation approach based on the maximization of the profile likelihood function. Monte Carlo experiments and two empirical applications to the size of US metropolitan areas and of firms in an Italian district confirm that the proposed method works well and outperforms two commonly used techniques. Simulation results are available in an online supplementary appendix.

Highlights

The lognormal and the Pareto distribution are common models for various phenomena in nature and in the social sciences
After estimating a lognormal–Pareto mixture with a known threshold via the EM algorithm, we exploit this result to develop an unsupervised tail estimation approach based on the maximization of the profile likelihood function
In this paper we develop a framework that explicitly takes into account the lognormal–Pareto mixture distribution of the population

Summary

Introduction

The lognormal and the Pareto distribution are common models for various phenomena in nature and in the social sciences. Most investigations in the literature focus on two related issues: using hypothesis testing or goodness-of-fit techniques to discern whether the tail of the distribution is lognormal or Pareto, and estimating the parameters of the latter. Our likelihoodbased procedure starts by assuming that xmin is known, and estimates the parameters using the EM algorithm (Dempster et al 1977) This hypothesis is relaxed, and estimation is performed by maximizing the profile likelihood function based on all the observations. In practical applications, we point out that the commonly used methods for detection and estimation of a Pareto tail implicitly assume no overlap of the two distributions and do not use all the available information. 2 we detail the EM algorithm for estimating the parameters, the profile likelihood approach employed when xmin is unknown and the llr test for the null hypothesis of no Pareto tail.

A mixture-based approach

Background: the EM algorithm

Known threshold

Unknown threshold

Testing for a Pareto tail

US metropolitan areas

Firm size

Conclusions