Hill Estimator Research Articles

A New Class of Reduced-Bias Generalized Hill Estimators

The estimation of the extreme value index (EVI) is a crucial task in the field of statistics of extremes, as it provides valuable insights into the tail behavior of a distribution. For models with a Pareto-type tail, the Hill estimator is a popular choice. However, this estimator is susceptible to bias, which can lead to inaccurate estimations of the EVI, impacting the reliability of risk assessments and decision-making processes. This paper introduces a novel reduced-bias generalized Hill estimator, which aims to enhance the accuracy of EVI estimation by mitigating the bias.

Identifying and analyzing power-law scaling in two-dimensional image datasets.

Power-law distributions provide a general description of diverse natural phenomena in which events with a logarithmically increasing size occur with logarithmically decreasing probability. However, experimentally derived correlated two-dimensional information is often difficult to cleanly interpret as discrete events of defined size. Moreover, physical limitation of techniques such as those based on scanning probe microscopy, which can ideally be used to observe power-law behavior, reduce event number and thus render straightforward power-law fits even more challenging. Here we develop and compare different techniques to analyze event distributions from two-dimensional images. We show that tracking interface position allows the associated scaling parameters to be accurately extracted from both experimental and synthetic image-based datasets. We also show how these techniques can differentiate between power-law and non-power-law behavior by comparison of Hill, moments, and kernel estimators of this scaling parameter. We thus present computational tools to analyze power-law fits in two-dimensional datasets and identify the scaling parameters that best describe these distributions.

Quantitative analysis of the intensity distribution of optical rogue waves

The field of optical rogue waves is a rapidly expanding topic with a focus on explaining their emergence. To complement this research, instead of providing a microscopic model that generates extreme events, we concentrate on a general quantitative description of the observed behavior. We explore two complementary top-down approaches to estimating the exponent describing the power-law decaying distribution of optical rogue waves observed in supercontinuum generated in a single-mode fiber in the normal-dispersion regime by applying a highly fluctuating pump. The two distinct approaches provide consistent results, outperforming the standard Hill estimator. Further analysis of the distributions reveals the breakdown of power-law behavior due to pump depletion and detector saturation. Either of our methods is adaptable to analyze extreme-intensity events from arbitrary experimental data.

Improvements in the estimation of the Weibull tail coefficient: A comparative study

The Weibull tail‐coefficient (WTC) plays a crucial role in extreme value statistics when dealing with Weibull‐type tails. Several distributions, such as normal, Gamma, Weibull, and logistic distributions, exhibit this type of tail behavior. The WTC, denoted by , is a parameter in a right‐tail function of the form , where represents a regularly varying cumulative hazard function with an index of regular variation equal to 1/ . The commonly used WTC‐estimators in literature are often defined as functions of the log‐excesses, making them closely related to estimators of the extreme value index (EVI) for Pareto‐type tails. For a positive EVI, the classical estimator is the Hill estimator. Generalized means have been successfully employed in estimating the EVI, leading to reduction of bias and of root mean square error for appropriate threshold values. In this study, we propose and investigate new classes of WTC‐estimators based on power of the log‐excesses within a second‐order framework. The performance of these new estimators is evaluated through a large‐scale Monte Carlo simulation study, comparing them with existing WTC‐estimators available in the literature.

Extreme Value Index Estimation in the Extreme Value Theorem under Non-Linear Normalization

The primary goal of this study is to expand the application of the extreme value theorem by developing the modeling of extreme values using non-linear normalization. The issue of estimating the extreme value index (the non-zero extreme value index) under power and exponential normalization is addressed in this study. Under exponential normalization, counterparts of the Hill estimators for the extreme value index estimators under linear normalization are proposed based on the characteristics of the extreme value index, threshold, and the data itself. In addition, based on the generalized Pareto distributions, more condensed and flexible Hill estimators are proposed under power and exponential normalization. These proposed estimators assist us to choose the threshold more flexibly and getting rid of data waste. The R-package runs a thorough simulation analysis to examine the effectiveness of the suggested estimators.

Estimation of Extreme Risk Measures for Stochastic Volatility Models with Long Memory and Heavy Tails

Financial data, such as returns on investments, typically exhibit some non-standard features: long memory or long range dependence (LRD) and heavy tails. Therefore, any mathematical model approximating the evolution of asset price should be able to generate these properties. This can be achieved through the use of a long memory stochastic volatility (LMSV) model. The focus is on estimation of Value-at-Risk (VaR) and Expected Shortfall (ES) for such models. While long memory has no effect on the Hill estimator of the tail index, in contrast it is shown that long memory affects the rates of convergence and asymptotic behaviour of the estimators of VaR and ES.

Flexible extreme value inference

We introduce flexible small sample modeling for extremes by introducing the new numerical characteristics of heavy tail. We illustrate in this article advantages of such flexibility. In this article, we show that we can obtain asymptotic normality of generalized Hill estimators by application of Karamata’s representation for regularly varying tails. Second order regularity conditions however better relates to Edgeworth types of normal approximations albeit requiring larger data samples. Finally both expansions are prone for bootstrap and other subsampling techniques. All existing results indicate that proper representation of tail behavior play a special and somewhat intriguing role in that context. The application of this new methodology is simple and flexible, handsome for real data sets. Alternative and powerful versions of the Hill plot are also introduced and illustrated on real data of snow extremes from Slovakia. We also demonstrate the importance of box-plot based techniques for small samples.

Estimation of the Peak over Threshold-Based Design Rainfall and Its Spatial Variability in the Upper Vistula River Basin, Poland

The proper assessment of design rainfalls with long return periods is very important because they are inputs for many flood studies. In this paper, estimations are performed on daily design rainfall totals from 16 meteorological stations located in the area of the Upper Vistula River Basin (UVB), Poland. The study material consists of a historical series of daily rainfall totals from the period of 1960–2021. The peak over threshold (POT) method is used, and the rainfall depth over threshold is assumed to follow the generalized Pareto distribution (GPD) with parameters estimated from Hill statistics. Alternatively, the competitive method based on annual maxima (AM) is applied. The theoretical distribution of AM is assumed to follow a theoretical distribution function selected by using the Akaike information criterion (AIC) from a family of seven candidate distributions, the parameters of which are estimated by using the maximum likelihood method. The two methods are compared by using the root mean square error (RMSE) and the mean deviation error (MDE) criteria. It is found that the POT-based method with GPD and Hill estimators outperform the AM-based method when considering the highest rainfall events. The confidence intervals of the design rainfalls, derived by using the Monte Carlo simulation method, reflects their large spatial diversity across the UVB. It is shown that the station’s altitude strongly correlates with the threshold, variance, and design rainfall depth of the GPD. This proves the advantage of the GPD with Hill estimates, namely that it can accurately reflect the spatial properties of rainfall and its variability in the UVB. Results can be applied in water-management applications related to floods.

Change-Point Tests for the Tail Parameter of Long Memory Stochastic Volatility Time Series

We consider a change-point test based on the Hill estimator to test for structural changes in the tail index of Long Memory Stochastic Volatility time series. In order to determine the asymptotic distribution of the corresponding test statistic, we prove a uniform reduction principle for the tail empirical process in a two-parameter Skorohod space. It is shown that such a process displays a dichotomous behavior according to an interplay between the Hurst parameter, i.e., a parameter characterizing the dependence in the data, and the tail index. Our theoretical results are accompanied by simulation studies and the analysis of financial time series with regard to structural changes in the tail index.

Probabilistic data self-clustering based on semi-parametric extreme value theory for structural health monitoring

Clustering is a popular and useful unsupervised learning method with various algorithms for applying to many engineering problems. However, some practical and technical issues such as severe variability in unlabeled data, necessity of determining the number of clusters, and utilization of an additional technique for threshold estimation in the problem of anomaly detection may limit applications of clustering algorithms to structural health monitoring (SHM). To address these challenges, this article proposes a novel probabilistic data self-clustering method for damage detection of large-scale civil structures by getting an idea from semi-parametric extreme value theory. The crux of this method lies in considering each unlabeled data as a local cluster. Accordingly, it is possible to deal with the major technical challenge of determining the number of clusters regarding most of the state-of-the-art clustering techniques. Using an unsupervised nearest neighbor search, each local cluster contains extreme values as negated minimum distances concerning representative data and ignores irrelevant data sensitive to outliers and any source of variability. With this strategy, the proposed method can address the major challenge of environmental and/or operational variability in SHM. A new anomaly score is then proposed to not only discriminate an abnormal (damaged) condition from a normal (undamaged) one, but also use it for estimating a decision threshold. This score is a high quantile of the selected distances within each local cluster estimated by the generalized Hill estimator under a semi-parametric peak-over-threshold technique. The key novel contribution of the proposed anomaly score against most of the existing scores is its great ability to simultaneously determine a damage index and a decision threshold within an integrated framework. Long- and short-term dynamic features (modal frequencies) of a concrete box-girder and a cable-stayed bridge are applied to verify the proposed method along with several comparisons. Results demonstrate the effectiveness and superiority of this method over some state-of-the-art techniques.

Scarce Sample-Based Reliability Estimation and Optimization Using Importance Sampling

Importance sampling is a variance reduction technique that is used to improve the efficiency of Monte Carlo estimation. Importance sampling uses the trick of sampling from a distribution, which is located around the zone of interest of the primary distribution thereby reducing the number of realizations required for an estimate. In the context of reliability-based structural design, the limit state is usually separable and is of the form Capacity (C)–Response (R). The zone of interest for importance sampling is observed to be the region where these distributions overlap each other. However, often the distribution information of C and R themselves are not known, and one has only scarce realizations of them. In this work, we propose approximating the probability density function and the cumulative distribution function using kernel functions and employ these approximations to find the parameters of the importance sampling density (ISD) to eventually estimate the reliability. In the proposed approach, in addition to ISD parameters, the approximations also played a critical role in affecting the accuracy of the probability estimates. We assume an ISD which follows a normal distribution whose mean is defined by the most probable point (MPP) of failure, and the standard deviation is empirically chosen such that most of the importance sample realizations lie within the means of R and C. Since the probability estimate depends on the approximation, which in turn depends on the underlying samples, we use bootstrap to quantify the variation associated with the low failure probability estimate. The method is investigated with different tailed distributions of R and C. Based on the observations, a modified Hill estimator is utilized to address scenarios with heavy-tailed distributions where the distribution approximations perform poorly. The proposed approach is tested on benchmark reliability examples and along with surrogate modeling techniques is implemented on four reliability-based design optimization examples of which one is a multi-objective optimization problem.

Random matrix analysis of deep neural network weight matrices.

Neural networks have been used successfully in a variety of fields, which has led to a great deal of interest in developing a theoretical understanding of how they store the information needed to perform a particular task. We study the weight matrices of trained deep neural networks using methods from random matrix theory (RMT) and show that the statistics of most of the singular values follow universal RMT predictions. This suggests that they are random and do not contain system specific information, which we investigate further by comparing the statistics of eigenvector entries to the universal Porter-Thomas distribution. We find that for most eigenvectors the hypothesis of randomness cannot be rejected, and that only eigenvectors belonging to the largest singular values deviate from the RMT prediction, indicating that they may encode learned information. In addition, a comparison with RMT predictions also allows to distinguish networks trained in different learning regimes-from lazy to rich learning. We analyze the spectral distribution of the large singular values using the Hill estimator and find that the distribution cannot in general be characterized by a tail index, i.e., is not of power-law type.

Asymptotic and finite sample properties of Hill-type estimators in the presence of errors in observations

We establish asymptotic and finite sample properties of the Hill and Harmonic Moment estimators applied to heavy-tailed data contaminated by errors. We formulate conditions on the errors and the number of upper order statistics under which these estimators continue to be asymptotically normal. We specify analogous conditions which must hold in finite samples for the confidence intervals derived from the asymptotic normal distribution to be reliable. In the large sample analysis, we specify conditions related to second-order regular variation and divergence rates for the number of upper order statistics, k, used to compute the estimators. In the finite sample analysis, we examine several data-driven methods of selecting k, and determine which of them are most suitable for confidence interval inference. The results of these investigations are applied to interarrival times of internet traffic anomalies, which are available only with a round-off error.

Extremal quantile autoregression for heavy-tailed time series

The conventional quantile estimator of the extreme conditional quantiles under quantile autoregression models is relatively unstable due to data sparsity in the tails. Extreme value theory provides a mathematical foundation of extrapolation to estimate extreme quantiles. However, the asymptotic distributions of existing estimators are often complicated making it inconvenient to apply for statistical inference. We develop a new adaptive estimation procedure based on a generalized Hill estimator of the extreme value index to estimate extreme conditional quantiles in autoregression models. We establish the asymptotic normality of the proposed estimators under some regularity conditions by applying the martingale central limit theorem. The asymptotic variances have closed expressions and can be directly estimated. Based on these results, we propose a procedure to construct confidence intervals for the extreme conditional quantiles. We also provide a diagnostic tool to check the heavy-tailedness of the distribution. Simulation studies and a real data analysis are carried out to demonstrate the advantages of the proposed methods over existing approaches.

Adapting the Hill estimator to distributed inference: dealing with the bias

The distributed Hill estimator is a divide-and-conquer algorithm for estimating the extreme value index when data are stored in multiple machines. In applications, estimates based on the distributed Hill estimator can be sensitive to the choice of the number of the exceedance ratios used in each machine. Even when choosing the number at a low level, a high asymptotic bias may arise. We overcome this potential drawback by designing a bias correction procedure for the distributed Hill estimator, which adheres to the setup of distributed inference. The asymptotically unbiased distributed estimator we obtained, on the one hand, is applicable to distributed stored data, on the other hand, inherits all known advantages of bias correction methods in extreme value statistics.

Estimating the Tail Index of Conditional Distribution of Asset Returns

Massive stock market failures in the past decades cast a doubt on the standard normality assumption of many economic models. Despite decent research on the non-Gaussian characteristics of many financial time series, the question of tail heaviness still remains open. We conduct diagnostic analysis on the conditional distribution of asset returns of small/large companies (Russell 2000 and S&P 500) to look for clear evidence on the presence of heavy tails. We employ extreme value (EVT) tools in order to estimate the shape parameter () of Generalized Pareto distribution (GPD) using a well-known “Hill estimator”. It turns out that the shape parameter lies in the interval implying that the conditional distribution of asset returns supposedly has finite mean and variance. We also find an evidence that the tail estimates experience structural breaks during 2008 Global Financial Crisis.

Box-Cox Transformations and Bias Reduction in Extreme Value Theory

The Box-Cox transformations are used to make the data more suitable for statistical analysis. We know from the literature that this transformation of the data can increase the rate of convergence of the tail of the distribution to the generalized extreme value distribution, and as a byproduct, the bias of the estimation procedure is reduced. The reduction of bias of the Hill estimator has been widely addressed in the literature of extreme value theory. Several techniques have been used to achieve such reduction of bias, either by removing the main component of the bias of the Hill estimator of the extreme value index (EVI) or by constructing new estimators based on generalized means or norms that generalize the Hill estimator. We are going to study the Box-Cox Hill estimator introduced by Teugels and Vanroelen, in 2004, proving the consistency and asymptotic normality of the estimator and addressing the choice and estimation of the power and shift parameters of the Box-Cox transformation for the EVI estimation. The performance of the estimators under study will be illustrated for finite samples through small-scale Monte Carlo simulation studies.

Improved Shape Parameter Estimation for the Three-Parameter Log-Logistic Distribution

The log-logistic distribution is widely used in different fields of study such as survival analysis, hydrology, insurance, and economics. Recently, Ahsanullah and Alzaatreh studied the best linear unbiased estimators for the location and the scale parameters of the three-parameter log-logistic model. The same authors also propose a shift-invariant Hill estimator for the unknown shape parameter. In this work, we propose a new estimation method for the shape parameter. We derive its nondegenerate asymptotic behaviour and analyse its finite sample performance through a Monte Carlo simulation study. To have precise estimates, we present a method for selecting the threshold. To illustrate the improvement achieved, efficiency comparisons are also provided.

Large degrees in scale-free inhomogeneous random graphs

We consider a class of scale-free inhomogeneous random graphs, which includes some long-range percolation models. We study the maximum degree in such graphs in a growing observation window and show that its limiting distribution is Frechet. We achieve this by proving convergence of the underlying point process of the degrees to a certain Poisson process. Estimating the index of the power-law tail for the typical degree distribution is an important question in statistics. We prove consistency of the Hill estimator for the inverse of the tail exponent of the typical degree distribution.

Handling missing extremes in tail estimation

In some data sets, it may be the case that a portion of the extreme observations is missing. This might arise in cases where the extreme observations are just not available or are imprecisely measured. For example, considering human lifetimes, a topic of recent interest, birth certificates of centenarians may not even exist and many such individuals may not even be included in the data sets that are currently available. In essence, one does not have a clear record of the largest lifetimes of human populations. If there are missing extreme observations, then the assessment of risk can be severely underestimated resulting in rare events occurring more often than originally thought. In concrete terms, this may mean a 500 year flood is in fact a 100 (or even a 20) year flood. In this paper, we present methods for estimating the number of missing extremes together with the tail index associated with tail heaviness of the data. Ignoring one or the other can severely impact the estimation of risk. Our estimates are based on the HEWE (Hill estimate without extremes) of the tail index that adjusts for missing extremes. Based on a functional convergence of this process to a limit process, we consider an asymptotic likelihood-based procedure for estimating both the number of missing extremes and the tail index. We derive the asymptotic distribution of the resulting estimates. By artificially removing segments of extremes in the data, this methodology can be used for assessing the reliability of the underlying assumptions that are imposed on the data.